GitLab Solutions Architecture, 4 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Audit Log forwarding in a GitLab Kubernetes Setup

Description

In this Video we are having a look how the audit logs can be forwarded from your GitLab instance. Please remember - GitLab Free only creates a very small amount of events. GitLab Premium has much more.

00:20 Documentation for Package installation
01:00 GitLab Helm Chart Architecture and Setup Walkthrough
03:15 Retreive the Audit Logs from the Cluster
05:05 How to forward the logs to your log system

For feedback please file an issue at gitlab.com (https://gitlab.com/gitlab-org/gitlab-docs/-/issues) and tag me @alex-dess

Links:
Deploy the GitLab Helm Chart:
https://docs.gitlab.com/charts/installation/deployment.html

Location of the log-files for audit events:
https://docs.gitlab.com/ee/administration/logs/index.html#audit_jsonlog

A

So hey and welcome to that quick tutorial on how to um get the audit logs from a premium version um uh of gitlab deployed to kubernetes. So why we're doing that is some because I have a customer request and it's a very valid request, because into our documentation, we're stating how to retrieve the audit Json lock um and how you can do that on a Linux package installation.

A

So when you're using the omnipus package, our official ones or even if you're self-compiling, there's there's just simply a log file which you can yeah set up to forward to your normal logging system, whatever that might be so, however, when you're deploying to kubernetes, um it's not not clear how you can achieve that, and therefore um I just like want to walk you very quickly through my setup and my setup is straight forward. So that means I just used the gitlab Palm chart, which contains all the components right for production.

A

You will have more a cloud native hybrid setup. This is what we're running on gitlab.com as well. So that means that the the components which are holding persistent so, um uh for example, redis or kitelli itself, and that will not be deployed in kubernetes but for test setups. That's a very good and easy version to do that, and the web service, where we will focus on, because this is the one creating the audit, lock events. This is like a scalable component which will be deployed in kubernetes even into a cloud native, hybrid setup.

A

So therefore, it's applicable so just to show you um how I installed gitlab very easy. This is all I did um and um then you can retrieve uh the um the root password with this command, because it's stored in a secret and, as you know, like Community secrets, are basically just like base. 64. got it, so you can retrieve that um yeah.

A

So what you need to do, in addition, is um when you deploy that, um depending on the cloud provider of your choice, um the chart will deploy an Ingress controller and this Ingress controller will get an external IP. So what I've done is I went to my domain, which is Dot getups.art and I, set up a wildcard um to forward to this IP address and then the Ingress controller will handle accordingly.

A

So you see, there's cert manager deployed as well, so we will have a nice certificate issued, um so you will not get a certificate warning when you're using that. So, let's first of all, let's, let's first of all, have a look at our gitlab instance.

A

So and I can I can just do that like this, for example, then you'll see kind of the ingresses which are created, and you see there is kid left.kitups.art um and I'm just going here to gitlab the GitHub store art and you see, there's a certificate and I am already logged on here.

A

So let's see how we can retrieve the the audit log for audit events in here for the sake of easiness I just deployed all the parts um to the default namespace, so you will have a separate namespace, most likely for a mirror applications where they get deployed. But you see here that all the pods from gitlabs who giddly redis like I, said self-contained are deployed in the default namespace.

A

So let's have a look. What we want to look at is the web service. Here you see that we're kind of like running two instances, so our two two children irrelevantancy- that makes a lot of sense, so it enables scalability and like if you upgrading, you will always um have a web service available in here.

A

So what we want to do um is we want to have a look at the locks of this web services, which will contain this audit log and what I'm doing is you can see that uh I'm following the logs of a web service?

A

um It has multiple containers, so we're specifying the container web service and grabbing the audit Json lock here yeah. uh So let's do that for now.

A

Okay, uh there was something and I'm going to do that for the second part as well. So it seems that this is not handling our traffic, but oh yeah. So we see so this seems to be handling our traffic right now, and uh this is kind of the audit log. So we're we're we grab for that, and you see that um there is a message created.

A

um It's a verti info and it basically just said like okay root locked on, so this is kind of an auto demand, so we locked on with root in here. So so this is what we want to follow on. The good thing is here so from now on, you see this is just uh uh um a log message which gets outputted to to the standard out, so you can whatever lock system you're using.

A

If it's like flu in bed or fluently, you can very easily set up a config map and just forward that from uh the the web service deployment which you're having so so just to show you so very flexible approach, whatever you're using. So you see here's kind of our deployment um just for me very easily. uh What I've done just to wrap that up to show you. The locks are really um easy to forward to any kind of lock system.

A

So I deployed a gitlab to a Google kubernetes cluster here and therefore I'm, just like using fluid bit to for what the logs um to my Google log Explorer. But it can be logstash. It can be elasticsearch whatever you're using and um I. Just like searched a message and don't don't worry about the timestamp right now, because this is a message from yesterday.

A

um I just opened lockstream here and you'll see that um basically, you can then set up your filtering to kind of like show all the all the Json logs- and this was an administrator log on here as well.

A

um So uh thanks for watching um and thanks very much to my colleague Mitch, which appointed me um towards uh towards uh the pot which generating the locks here, if you have any questions, uh feel free um to ask them and um I'm looking forward to your feedback and if that's helpful, for you in this case.