GitLab Fuzzing, 13 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sam coaches Cindy on fuzz testing demos

Description

Fuzz testing
Associated presentation: https://docs.google.com/presentation/d/1YSOvlS-0wmxtdsEky5m3PijslFOLRUzppFJvQhMaFMo/edit#slide=id.ga0dc1640fd_0_1959

A

A

Thank you, okay, so that so the point I need to emphasize about the openssl is what now.

B

So this is just the regular version. We've not made any code changes to it to make it amenable for fuzz testing, um because one of the pain points people are going to be wary of is, if I want to do this, I'm going to have to do custom development on my source code base, which you do not have to do with fuzz testing, um so they could go download this exact version from the internet and reproduce our results very easily.

A

B

So really, there's going to be two steps in adding coverage grade fuzz testing to the project. The first one is that we have to tell the fuzzer how we want to run and fuzz the application, because the fuzzer is unique to our app and it's going to find business logic issues unique to our app. We have to give it that extra context and to do that we're going to build what's called a fuzz target, and that's in this handshake fuzzer cc file.

B

Let's go ahead and open this this one up, so a lot of this file is boilerplate. That would be very familiar to an open, ssl developer. um Essentially, what we're doing here is we're setting up the tests to be able to run successfully the way that we interface with the fuzzer. Is we implement this one function? This love, fuzzer test, one input function and what this is doing, the fuzz engine is going to call this over and over and provide random data every single time it calls it that's going to be different each time.

B

That's what this data field is, and then it gives us the length of that field in this specific fuzz test. We have to set up our environment, and this is standard openssl boilerplate. This would be very common to an openssl developer or you could even copy it from one of your unit tests.

B

The function we actually want to test, though, is this bio underscore write function, so all we have to do. There is rather than provided a piece of mock data. We just provided the random data that the fuzzer is generating, and it's just going to call this over and over and over again, and that's all we have to do.

B

We also do some cleanup at the end, which again is open, ssl, boilerplate,.

B

Any questions on that one before we go to the next piece.

A

B

um The second part is now that we've written our fuzz target. We have to tell gitlab to run it as part of our pipeline on every build. So for this we'll go to the ci yaml file and we're going to do two things.

A

B

First, one is working well.

A

I had a question about that kim, so we can't set that up in auto devops or in the we've now got that like assisted setup.

B

Not yet no it's something we're talking about, but nothing uh nothing today for that.

A

So so you have to go into the ciaml yeah yeah.

B

So we're going to first include a coverage fuzzing template. This is just going to define all the backend pieces that the job needs to run and then we're going to define our actual fuzz testing job in this case called heart bleed target, and what we have to do here is we have to actually build that piece of fuzz target code that we just wrote and that's what all of these lines are doing.

B

This is all pretty generic to open ssl and it's going to be different for every single application. But essentially what we're doing here is we're. Taking that fuzz target code, we had we're building in such a way that it can call the various functions that we want to test, and then the final line of every fuzz target is going to be this one on line 25, where we call gitlab fuzz and that's the actual fuzzer and we're just going to say, run and then give it the name of that fuzz target.

B

We wrote that handshake, fuzzer binary. That's all we have to do.

A

B

So once we've done that we'll make a commit run a pipeline and let's take a look at the results that we'll get so in this case, you can see I've run my unit test as part of my normal testing. The fuzzer heartbleed target ran, and so we knows they found something. So if we go to the security tab, we can see that I found one vulnerability.

B

This heap buffer, overflow read and so we're given a bunch of different information here about the crash address of where it crashed in our target binary.

B

The type um we're also given some information about the function, names that were in the stack trace where the crash occurred, as well as the actual full stack trace here- and this is the really detailed information from the job log that a developer would use to figure out where exactly the crash occurred.

B

If we scroll down here- and this is the part you'll want to highlight the demo- is this tls1 process heartbeat item? This is where the actual crash occurred in the code, and this is where the heartbleed vulnerability gets its name from, because it's a vulnerability in that heartbeat function. They called it heartbleed.

B

And so you can see that in this crash state here, as well as all the detailed information from there, you know you can create issues you can discuss with your security team dismiss them. The standard shift left things that we talk about.

A

So, what's so what parts missing? Because you've got the widget, so you just don't, have it in the mr pipeline.

B

Yeah so uh where's an mr, we could look at because, basically you know how, when you have an mri it'll say like security results and expand.

A

The fuzz results.

B

Are not put there so um so you.

A

Have to okay, so you just have to go to that security. Tab.

B

Yeah um one question you'll probably get uh is that one is someone will say you know the stack trace is great, but I need more information than this like the inputs that caused it. Those are all available in the job itself, as artifacts um there's a lot more info in the job log. But if you go into the browse option on the job artifacts area.

A

How did you get to that.

B

So we're on the pipeline here, if we just click on the job name, it'll be on the job artifacts on the right here and so we'll just browse through them. This is going to have all the really low level information that a developer might want um the corpus here. These are all the test cases that the fuzzer tried and these are going to be what they can use to reproduce the crash locally.

B

um No, I'm sorry, I told you the wrong thing. The crashes directory is the directory. They will use to reproduce the crash locally if they want okay, um so I wouldn't highlight this on the demo, but you will likely get a question about this and that's where that info is.

A

A

A couple questions: um what if I just skipped how to set it up and went straight to showing the results, because to me that's the most impactful part.

B

um Yeah that's an option. I I think it'll really come down to the timing like which parts you want to really focus on, um because I mean I agree. I think this is really the like. Oh we, we found this with very minimal configuration.

A

Exactly because the other stuff is a little bit in the weeds, I'm afraid, because what what I want to show is we're making fuzz testing simple and bringing it to. You know the everyday user right and I feel like this. The setup part counters that message. Yeah.

B

Well, so that's the that that's the thing, um because we have to emphasize to people that fuzz says it's easy to get started with, because that's where classical fuzzers fail right, I feel like. If you only show the results, people will be like yeah, but you probably have to spend a week setting it up right. So the intent with the early part of the demo is to say this is really simple.

B

You know I walked you through it in a few minutes versus a week, um but that said, there probably is a way to go through it more quickly.

B

Maybe what you could do is just briefly go through, like there's only two steps to doing this. You have to provide a fuzz target which is just how to run your application in the fuzzer and then add it to your ci yaml file and then move on from there.

A

Yeah, I think that would work yeah.

B

Yeah that I think that would probably get the get the point across.

A

Okay and then um on the slide deck did you have? Do you have that duck candy um yeah this.

B

A

The part with the orange boxes um you did that on commit. Didn't you.

B

A

um Can you just quickly walk through how you talked about that because, well, just as a refresher, yeah.

B

um So in this example, we want to talk about an a conceptual or a hypothetical application. Let's say that it takes a json document as an input processes it and does some sort of service like producing a report.

B

So as an example, you might provide a json document that is requesting a balance for customer number, five and you're, giving it some sort of auth token, that's unique to you when you're logged in the app processes it you get it successfully and you can do whatever you want with the report because you're doing good testing your application would reject erroneous cases like when you ask for an aaa type of request.

B

These are the type of random things that the fuzzer will try as part of providing those random inputs to your app.

A

And that's just.

B

A

That you did like input, validation, right.

B

Exactly okay, because the the thing we want to highlight here on this slide is the aaa is something the fuzzer might have tried, but because the app was well written and has tests it returns. This error case, as expected.

A

B

Another example: maybe the fuzzer tries to provide an invalid customer id and again we have error checking in place. So it doesn't matter, um but let's say the fuzzer finds one case where it provides a valid customer id, but doesn't change the auth token and your app just returns a report as expected.

B

That's a major security issue, because the auth token I provided the fuzzer provided was unique to customer five, but it got the results for customer one. um So, even in an app that has qa and testing in place, the fuzzer found an edge case that wasn't covered previously.

A

A

All right, so um that all makes sense, I may add a slide in there for peach and buzz it. I took that out. I might add it back in.

B

um I thought you had one one of the acquisition slides. You mean, I thought we had one in here.

A

Yeah, it was in there and I took it out, but I think I'm gonna put it back in.

B

Just a note when I've gone through both this slide and this slide.

A

B

Have been time sinks uh that take a little bit longer than I expected so.

A

B

Recommend either running just this or the demo.

A

B

For whatever type yeah.

A

Well, tell me how you want.

B

A

About that one and the 18, if those are what what gets you, what takes up so.

B

Yeah, so in this one, the the thing that takes the time is, you have to set the stage kind of for like what even is this function? Why are we looking at it and we have to talk through where we have this parts complex function? That takes an input we can see in the logic, though, there's off by one error, because it's checking for seven letters here, but it's only looking at a length of a sixth string.

B

The fuzzer is what would generate that sort of random data and eventually find that test case, um and that pretty much is the whole presentation of this slide is redundant when you also show a full demo of the fuzzer, so.

A

B

I I did a dry run for uh the cab with both this slide and then going into the heartbleed demo, and it was a lot of time and a lot of seeing the same thing. Okay, if that makes sense. um That said, if you don't do an api fuzzing demo, I think this one's actually really good.

A

Okay, so how do you talk through that? One.

B

So in this one I don't know, if there's animations, I might cut out the other three just so they're, not visually distracting. When you first see this.

A

B

And then do like a click-through animation of it, but essentially what we've got is a recording of a request to a web application of some sort. If we look in the top left, we can see the api definition.

B

It's defining an api endpoint at the root on line 19 that takes a get request, takes a variety of parameters such as the name which is on line 30.

B

and when we provide a valid uh request, such as the one pictured on the bottom left, where we're saying the user is dd first mic last smith, password hello, you know we'll get a uh sorry yeah when we provide this value request, we'll get a valid sort of response.

B

But what we're going to do is give this to the api fuzzer and it will say: okay. This is a known good type of request. I'm going to take this and I'm going to mutate it. So in the top right here we can see the fuzzer change, the username from dd to percent 27., and then, when we give that to our application, it responds very differently- and in this case on the bottom right, it actually crashes.

B

The server with an internal server error, so this is an example of how the api fuzzer takes known, good ways of interacting with the app either through an open api file or a recording of traffic and can mutate them to find those sorts of edge cases. We're looking for.

A

B

Cool yeah, so I like this slide a lot from a conceptual standpoint. I I think it's busy, though so I mean, if you like, did the you know one at a.

A

Time kind of thing I think.

B

It might be easier to follow.

A

Yeah, I agree and so show me, can you show me where, on the security tab, the api results are.

B

Yeah, so let me pull this one up.

B

I find what my autocomplete wants to show, um so this is the repo I added you on to so let's go to.

A

Looks very different from anything else I've seen so I was not sure where to go. This.

B

Is actually a cool gitlab feature? This is our open api. Rendering like this is all driven off of a json document. There's no like custom, markup or anything.

A

Oh wow, that's pretty cool.

B

So we're looking at this file just a json document and git lab auto pretties it up. It's really cool.

A

B

A

Let's go look at the pipeline.

B

This one seems good, so the api test testing is actually under our test. Tab for right now.

B

So we're actively working on pulling this into the security tab. The reason we did it this way is because peach from the acquisition already had json output, and this is rendered directly from that. So there is no uh or j unit output. Excuse me, so there is no net new development for us to get it in this form. So now we're working on the dev work, to put it in security.

A

B

So each of the fuzzing runs will report an api buzzer job, the duration, notably it found six things under the errors column. So if we click on that, it'll give us detailed info on each of them.

B

So it'll tell you the high level name of what it found. In this case it was a fuzzing injection on this api endpoint, and then it gives you a lot of information about what it found, what it means, as well as a detailed description.

B

So like you can look up the cwe and the owass, it has a human readable like just telling you what things are so this is actually really cool. um The part you want to highlight, though, uh actually I don't like this one, so I would scroll to. Is this the second or the third one.

A

So this is all successful together.

B

A

This is all six of them appended together.

B

Correct yeah, I would look at the third one in the list because the others require you to dig a little bit too deep for a demo for it to click with people. But what's going to really jump at people? Is this mutator part where the fuzzer saw? Oh, you tried value 2, I'm just going to try 65576 and see what happens, and in this case we have an internal server error.

B

And so then it's going to give you all the original stuff that it recorded um what it received when it you know, got that back as well as a bunch of other information that the devs could go ahead and you know figure out what went wrong.

A

Okay, yeah very cool yeah. I can show that okay, um but this does a good setup of what it is they're going to see. If I can keep it real, close and short.

B

Yeah, I think the other one. You have to caveat it so much because it's not in the security tab and it's kind of a different workflow and different screens like the coverage. Fuzzing is solid. I think this diagram gets the concept across and hopefully entices them to. Let's talk after the conference kind of thing.

A

No okay, um well, hopefully the q. A is easy, we'll see! That's the part that scares me. They ask too many questions.

B

I I'm sure you're gonna do just fine cindy you'll be great.

A

Well, thank you very much for for doing this. I appreciate it. I need I I thought about punting to see if you wanted to take it, but I thought I really need to learn this. So this will. This is the force function to make me make.

B

A

B

um If you've not seen this document in a while this fuzz session faq that we did a while back, this might be worth a read for some of the sorts of questions. You'll get.

B

um We originally targeted to sales people, so some of it's more like positioning sales, but there might be some technical bits in there that could come up.

A

Okay, that's helpful. Thank you. I've forgotten about that.

B

um When is the the presentation I'll do some other thinking? If I can think of any other questions, you might get.

A

um So I've got to record it today, but the.

B

A

Yeah but the um session is the 20s it's tuesday next tuesday, so the q, a with the live q, a is next tuesday.

B

I mean, if it'd be helpful, I'm happy to you, know, block some time off and sit in the chat room.

A

B

To help with that.

A

That might be good, just as my uh savior.

A

um I'll ask joe to he's using some weird uh meeting thing that I've never encountered before, but it was super simple, but um he'll probably have to send you a link for that um yeah. Okay, cool! Well! um Thank you very much. What I'll do is, after I record it I'll, send it to you and uh so that you know what what I said, what we're covering yeah yeah all right! Well, hey! Thank you again. I really really appreciate your help.

B