GitLab Observability, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Observability (Opstrace) local tracing demo

Description

This is a demo of deploying GitLab Observability (Opstrace) locally and using a test application to send and query traces.
This is using our new Golang operator code, GitLab auth, ClickHouse backend and an early version of our Observability UI (Grafana fork).

A

All right, so I'm going to show how we're going to deploy an instance of the current gitlab observability tooling, uh and to do that I've checked out a version of the upstream repository. I did a git pull someone in the latest. Yes, I'm up to date. um First things: first, we need to do a local cluster. There's a so! Let me see I'm going to go to the quick. I know these by heart, I'll, do them by heart and there's a quick start for anyone to do these so basically, first sorry mac.

A

So basically, first we're going to do make.

A

We're going to create a do, make kind, because we're deploying this on a local cluster, so I've already set up my mac and I've already given enough um enough ram and enough cpus for this cluster to work, but it's already that's also described in the click start.

A

The other thing is, I think, the whole end-to-end system on my machine will be a bit quicker than if somebody tries to replicate this, because I don't have to pull any images anymore or anything like that, so yeah. So right now it's starting the kind cluster once the kind cluster is started, I'll have a local kubernetes cluster to be able to deploy, to deploy, build and deploy.

A

This obstrace stack against it.

B

So, let's wait for a bit.

A

B

A

Okay, now we have a working cluster. We can check just to make sure up first time, always a bit slower all right and then we are going to uh sorry.

A

I cleared my screen and then I'm going to do a make, deploy, make deploy, is going to build and deploy the system to, and so that means a whole build and deploy uh what's going to happen is basically, we have a um we're building an operator that operator uh once built, gets deployed to the uh the kubernetes cluster and that operator will then deploy the rest of the infrastructure.

A

So that's what's happening here.

A

So since I did a git pull, I have a couple of fresh dependencies and now the build is done so we can see that it's applying and running applying things to the kubernetes cluster.

A

So everything that's getting created right now, like what you can see, especially in the part here. This one is: we've created, what's called custom resources for the uh this cluster, so we can now deploy tenants. uh We can now deploy dashboards. We can now deploy there's a bunch of different systems that we can deploy to one cluster. So now that all of this is done and deployed, the next step is to.

A

The next step will be to create a config file that the config file is here. I've already created, it's called cluster yaml. I've also created inside of my org inside of my own. So now I'm sharing my secret, but that's fine, I'll change it right after this video, but basically inside of my profile here, I inside of api, I've created an api where's api.

B

Api, where is api.

A

I don't remember applications here so in applications, I've create now it's a bit slow.

B

Right when I wanted it slow, it's okay,.

A

Okay, so I've created an application. It's down here. It's an application, called ops trace I've added a callback. Everything is in the quick start like like I explained, and that gives me the cli, the uh that gives me a client id and a client secret to put in there. um This shared secret is just a random string. Right now will be used in the later future. So don't worry about that one. So now that I have this cluster yaml, this is basically the definition of a cluster.

A

When you put this inside of this kubernetes cluster, a new cluster will get create a new austrace cluster will get created. So basically we're going to.

B

A

We're going to control apply this yaml file, so, what's doing so as soon as we've done that see, it's created one thing, which is something called the dev cluster, which is what I've currently deployed now we can actually wait and we're going to basically wait until this cluster is ready. That's one of the nice things of custom, resource definitions and operators. You, basically the operator, tells you. When everything is ready, you don't have to go hunt for things.

A

We can take a look because we're curious. uh We can do a get pod and we can continuously look at the output of what's happening here and you can see that the cluster is getting created, see these this. All of this is the the basics that are already installed inside of the uh the cluster that we did when we did make deploy. So it's all our operators and a few other things like a development postgres, and things like that.

A

We can see now that we have our first one running here.

A

And now the pods are getting created, so this is the longest part of the setup. I would say this is the part where our code is actually getting downloaded deployed to the system and uh deployed inside of the kubernetes cluster.

A

So um yeah it's going to be a few minutes.

A

So if this was a live demo, that'd be questions right now.

A

Do you have a question nicholas since you're here.

C

Yeah, I was trying to think if I have any any questions right now um so so this was not local. How would it be? I guess, how would this.

A

Exactly the same thing, so we used a kind cluster because that's easier to develop locally and everything, but if you deploy any a kubernetes cluster in the right configuration, which is what our terraform code does right like, uh then you can deploy the same system and code um to that uh cluster and run it there right like uh there's, there's not a big difference.

A

More and more things are getting deployed. There's a lot of dependencies right, like redis is used and other things like that things could gets much faster because right now things are spun in sequence: it's easier for debugging, it's easier for development. In the beginning, one day, when we're more confident, we can start spinning a couple of things in parallel.

A

Take a look, maybe everything in theory: you can spin up everything in parallel and it starts. This is how our our previous code worked, but um this is a better approach right now for development, like we're gonna find bugs faster. This way.

A

Just a few minutes, and once we're done once this is done, we'll be able to have a bit more of an interactive part to the demo.

A

So yeah this is clearly separated in two parts. Right like this part of the demo, is the part where I would say the operator, the the the person the who is going to deploy the system deploys it right and we want to make it much much easier than all of these commands right, like I've done, make commands and queue control commands. That's not what our customers are going to do in the end. Right, like that's, that's important to understand right.

C

Yeah, so so, so I guess, how do we imagine kind of automating away those steps or like what would you imagine.

A

About most of this can be generated right. What did I do here right? I ran a couple of cube control commands to deploy a couple of operators, so those are manifests that we can deploy and kick off the deployment from terraform from git lab. There's, there's plenty of places.

A

We can kick off this deploy these deployments from the I'm pretty sure uh the uh the engineers are thinking about this quite a bit already, but then other things that we can do are um so kicking off the deploy from gitlab uh that uh or terraform that's the main thing, um oh yeah, and then this this.

A

This thing that I went to and when I did the the the oauth secrets and everything that could also be done behind the scenes right like there's, no reason somebody who's installing gitlab shouldn't be able to click a few buttons to say I want this deployed and then would fill out a form saying uh this is where my kubernetes cluster is here's a here are some credentials to go there or if they don't even have a kubernetes cluster like kick it all off with our with our integrated terraform, deploy right like there's, no reason somebody should have to click anything and do anything manually uh all the way to deploy, and we know it works because we've done it before right, like the the pre, like the previous demos of upstream and the code, pre-acquisition was like that right like uh end to end: no, no, no human, all the way to just clicking into login.

A

That's the experience we want to bring back, but from gitlab right. Okay, okay, are we looking here? I think we're pretty good. Let's see, yeah we're pretty good look uh conditions are met um the cluster's up right, like that's it. It's that simple. uh Now we have a running cluster, but we don't have any um observability to ui to go to basically right, like we don't have an instance of the observability ui running for our group.

A

So let's start associating things- and that's done like this, so I chose this group here in which I am an admin. Okay, so I'm an owner of this. I can use this group. This is not the this. Is an old ops trace group I'm going to go to basically on local.

A

Oh, I need to do some forwarding nope.

B

A

It's right there we go.

B

There you go sorry.

A

There we go so again. These things are going to be nicer. This is mainly a demo, but you can see. Gitlab observability is not enabled for this namespace.

A

Okay, like usually you would not have the localhost url, obviously, but the nicer one, I'm going to click enable now we're just going to wait here. This again can be made faster, but to be fair, this happens once right, like you, can.

C

So what exactly is happening with in this loading sequence here.

A

So the loading sequence is mainly for, uh like demo purposes,.

B

A

Us being patient, what's happening behind the scenes, can, as usual, be looked at when you look at what pods are getting deployed to the kubernetes and the pods are basically the uh they are the the pods that are specific for that group. There are only for that group where the traffic for that group goes through, so there will be, for example, a pod deployed with uh the um gitlab observability ui right, like uh not just one pod, uh and then there will be uh like, as you see uh like a an operator for the tenant.

A

So what happens? Basically is we have an operator that then launches another one? The tenant operator and the tenant operators is responsible for one group, one tenant and that deploys the rest of the stack, an open, telemetry endpoint. You can see it here as well right, like jaeger, all the pieces that are specific to a user or not so much a user, but a um a group are getting deployed and that's what we're waiting on we're waiting on these things to run, and uh they should be up in a second like yeah. Let's just wait.

A

C

Is this is on the timer.

A

But as soon as see like as soon as we have all the pods running, we'll be happy so, and I think we're almost there so argus deployments is our ui. uh Jaeger is the jager endpoint and so on. Right, okay,.

B

Okay should be good, let's see.

A

Not yet soon soon, soon there- oh, let's see like once you've done it a few times. You know the timings. uh It's uh also you can look at the their hints right when you look and you know what to look for here. um Okay, now we have. uh This is our uh grafana fork. uh This is uh still says grafana, but that's gonna. Do people are going to be work work this over now we're going to add. First, we need to add an api key.

A

Why do we need to add an api key? Well, let's give me a second here we're going to call it test.

A

That is because we are actually starting some kind of simulator here. So give me a second here one. We don't need cube, control port, we're gonna forward.

B

A

This is all everything I'm doing right now: proxies forwarding everything, that's just because I'm actually uh doing a test where I'm going to deploy a website, that's emitting traces, and then I want to be able to go there and see the traces right. So the part that I'm doing right now is a docker compose that I'm deploying that is going to do that. So compose this token that I just got is needed by our demo application. Oh sorry,.

B

I'm going to start again so token.

A

I am still not used to all the mac short like keyboards, like I'm still doing all my linux things and then it blows up in my hand when I do use this okay and there we go. We just start this up. So when we do this docker compose up, we can see that we've, given it the namespace and the token, um and once this docking could pose up it's going to create traces when I visit the website.

A

So let's go there first, we'll do that later. Here is the demo website. The demo website creates traces when I click on these things right, like basically right, like that's.

C

A

That's all this website does now. We want to go look at them inside of the ui. To do that. First, we don't have any like. If you just go to explore, there's nothing. We don't have a data source, so let's go out of data source if you're familiar with grafana. This is not new again.

A

Adding a data source is something that we're doing here during the demo, but our system will do automatically right, like no user should have to put a data source of something uh that is basically already uh there right like it's deployed by us. Sorry, that's what it means. Why is this being annoying all right? That's fine! Okay, uh the url for the data source! Oh, that one is a complicated one that I need to copy paste.

A

Where was it, I forgot about that one, it's in the quick start. I can.

B

Get it okay, I'm going to stop. What am I going to stop here.

B

A

B

Gonna create a new one.

A

Can you still see my new screen.

C

Yeah, I think so: okay.

B

B

B

A

Oh, it's right here, great all that for nothing.

B

A

B

A

This quick start is available in a nice, readable format. Of course,.

B

There we go and there is the url http localhost in the above group. We open.

B

Oh, that's, not the one.

A

ah Split up sorry.

A

It says locally because there's already there's some other dogs there we go. This is the local version. This is the what I just did with all this docker compose and then data source tab.

B

There why ah there it is.

A

Yeah you'll see why I can't remember that one and why nobody should ever have to remember that one, because this is really some internal url, where.

A

We have to know naming scheme and.

B

A

B

A

B

A

Okay, green is good, explorer jaeger traces customer get, and this is the click that I did at a few minutes ago. So that's a trace for the last click. I did basically right on the system, so you can see and the traces inside of the ui it's stored in click house and so on right like so what happened once you click a trace, gets created, sent through open telemetry through open, telemetry, then stored in click house.

A

Then, once you go once you go to here and you do a query, it goes and retrieves it in click house and shows it with the jager plugin inside of the grafana ui and that's it. That's. The whole show for now error tracking coming soon.

A

Cool all right I'll, stop the recording.