GitLab Compliance Group - Product, 7 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Streaming Audit Events 2022-05 Walkthrough

Description

Documentation: https://docs.gitlab.com/ee/administration/audit_event_streaming.html

Epic Link: https://gitlab.com/groups/gitlab-org/-/epics/5925

A

Hi, my name is dennis tang. I am the full stack engineering manager for the compliance group under the managed stage and today we're going to be walking through audit event streaming um why you might use it how to set it up and what the events look like once you perform user actions and you start receiving events on your http event collector so streaming. Audit events is really useful for being able to receive all the audit events happening under your group, and so it allows you to have full access without going through the ui.

A

But it also, more importantly, allows us to be able to audit events that are typically high volume, so, for example, get repository actions if we were to store that in our relational database that can cause uh degradation in terms of performance and so by um asynchronously streaming. These odd events, in the background, um we're able to collect a lot more data and be able to audit more events that are typically high volume, and so in this example, I'm going to be walking us through our my local development environment.

A

The steps will be the same, whether it's on.com or your self-managed instance, um and so we'll go ahead and get started.

A

So what you want to do is you'll need to be an owner for whatever group you want to set up streaming, auto events on once you go to that group in this. In this example, we're going to be using commit 451, you're, going to go over to the security and compliance menu and then click on audit events.

A

This will just take you to the group audit events ui, where you can generally see the audited events happening or the actions being performed in your group, but there's also a second tab here now called streams which you can then use to add the endpoint for your http event, collector, and so in this demo, I'm going to be using requestbim.com.

A

You can use whatever you like pipe dream request, catcher again, if you're actually setting this up for a proper data collection application like splunk, then you would just use that endpoint that the http event collector would provide. um So in this case, I'm just going to create a new request bin. Just so we can analyze the events in real time without having to set up anything a little bit more complex.

A

Then we're going to go back to the ui and add this endpoint and that's it at this point. Any audit events that are performed will be streamed over to that uh collector, and then we can analyze the events you can as our documentation states. You can also set this up by graphql.

A

We have commands and document documented examples of how to create list and delete these, but for the purposes of this demo, we're just going to stick with the ui. So um now that we've got the destination set up, we can go into a repository. I'm going to start with the example of downloading the repository via http.

A

Excuse me downloading it through the ui, we're going to see what that looks like so you that download finished. We can take a look at request bin and see that a new um event has come in.

A

So the first thing we're going to look at are the headers um anytime. An event is streamed to your collector. There is a header called the gitlab event streaming token, which contains a token unique to your destination.

A

You can use this to verify the validity of the events coming in and, for example, uh because it's in the headers. You can then scan that and verify it without uh diving deeper into the request.

A

um But if we look at the body we'll see some general information about who did what so the author, as well as uh the target or the um object that the action is being performed on and if we dive deeper into the details, we'll see that lori wolfson had started a repository download, pretty straightforward, and this is the custom message used for uh in ui repository downloads.

A

Now, what does that look like when we clone the repository and so we're going to start by using um excuse me, I'm going to start by using a ssh git clone example.

A

So if I clone it, I should be able to go back into pipe dream now and see that a new event has come in and if I go into custom message: here's where it's uh the unique identifier in terms of what what methods being used is uh shown here. So we can see that the get upload pack action was performed over ssh and so effectively a user has cloned the lab coat repository over ssh. We can also do the same for this.

A

We can also do the same over http or https, depending on how your instance is set up, and so we're going to go ahead and do that right now.

A

Cool, so if we go back over into pipe dream, you'll see that a new http event has come over and again the same action was performed, but this time over the http protocol.

A

um What we can do now is show what it looks like when we make a change and push that back up. So, let's say, for example, I want to update the title just for simplicity's sake, I'm going to go ahead and add this.

A

And we'll push this back up and again we'll go back back over to pipedream. New events come in still happening over the http protocol. In this case, the action has changed since we have this user. Lori rolson has pushed to this repository, so that basically covers get repository actions. um I can show what it looks like to create a deploy token and perform actions in that manner.

A

So let's go ahead and clean this up cool, so we're going to go a little bit further and set up a deploy token for this repository and show what that looks like and that'll give a different look at the audit events of what's happening when you're actually creating that deploy token.

A

um So I'll just use this example here I'll call it lori deploy token, you can define an expiration date and use your name of course, and then define the scope of that deploy token and, as that's being created, you'll actually be able to see the audit event come in as well. That deploy token was created so again. Anything that's normally already being audited is now being streamed over to your collector and so now. I can then use this these credentials to then clone the repository.

A

So I'm going to go ahead and clone that copy that username.

A

Paste that password again same deal, I'm just using the deploy token, and if we look over here we'll see the new event again just pointing to the user of the deploy token and that the action performed was over the http protocol.

A

Other examples would be then, perhaps going through the group and adding a new member, perhaps you've added a new member as a maintainer, and you want to you know, keep track of all those changes. So let's say I upgraded this person's role from developer to maintainer. That would come in and you can see that the access level changed from developer to maintainer for admiral marvin.

A

I can add a new user to this group as well. Let's say I added a new maintainer and that would also be streamed to your collector. So freda was added just now as a maintainer. If we go over to our collector they'll see that freda as a target was added as a maintainer by lori rolfson.

A

That's it that's streaming, audit events in a nutshell: we are looking at developing or iterating on this further by allowing you to define custom verification tokens allowing you to filter events before they're actually sent over to your destinations. I'll include a link in the description to our epic that covers all the functionality, we're looking to add there.

A

One other thing I forgot is that we're looking to add more um uh storage uh solutions in terms of being able to push these audit events to an s3 bucket, for example, and that's it so thanks for joining me, um that's streaming, auto events. If you have any questions, please feel free to reach out to me my username is dennisongitlab.com and thanks for watching.