GitLab Infrastructure Group, 12 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Touring a Jsonnet generated GitLab CI configuration

Description

Craig Furman provides an overview into how we utilize Jsonnet to generate a decently sized GitLab CI configuration.

This is primarily utilized: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/tanka-deployments

More about Jsonnet: https://jsonnet.org/

Other places where we utilize Jsonnet: https://gitlab.com/gitlab-com/runbooks

A

Okay, so in today's recording craig who has joined me today is going to give me a tour of how the tonka deployments repository builds dynamically. Its ci configuration take it away, craig.

B

Hi starbucks uh craig didn't prepare this at all, so be prepared for a bumpy road with lots of ums and ours. I'm just getting all my stuff ready. So I'll share my screen all right. Maybe I need to make this font size, bigger.

A

I would highly recommend bumping the font size since I plan on showing this into youtube.

B

Okay, so before just diving into the actual jsonnet, which I don't think will be very helpful in isolation, I should describe what tanker deployments is and does tanker deployments is a mono repo that uses grafana's newfangled-ish tanker tool to deploy various things to kubernetes.

B

We've got a bunch of workloads, we'd like to run such as like thanos or like woodhouse, sre helper tool, and most of these have at least one environment. You know, for example, we have a thanos store in every google project in which we run a gitlab environment.

B

um We have you know, test environments for things, staging environments and soon we'll want to move prometheus in here, which is another kind of cluster local thing like at least one per cluster um and the aims of this. The aims of this repo are to be able to share oh yeah. So if anyone doesn't know, tanka is jsonnet based it's actually uh and if anyone doesn't know what json is. Jsonnet is a simple one of these. Like configuration languages, it's a full-fledged programming language, but quite stripped back and oriented around serializing json documents.

B

It can also serialize like yaml and ini, but um it's essentially a programming language without a whole lot of standard libraries for things that you might expect like interacting with files and networks and all that and it's instead just oriented oriented around functional, transforms of data structures for.

A

But hopefully easier.

B

A

Say bash where oh, it's.

B

A

To do arrays and such.

B

Yeah, absolutely uh it's it. It is quite simple, but the documentation is sometimes like a little lacking. On the other hand, I'm sure we'll see that um right. So what are our objectives? We want to have um a bunch of things we deploy. So tank deployments is new ish. We haven't got around to moving tons of stuff into it. It's only got five workloads like some of these things like cisco, patch, evicted, pod, reaper and thanos.

B

These are components that we want in every cluster as with jaeger and then some things we have in fewer clusters, but but whatever we put them in one repo, because why not?

B

Because we don't want to reinvent the ci deployment wheel and use things like gitlab ci includes, which can be a bit like annoying to sync up between lots of different repositories, and we want the opportunity to frictionlessly share jsonnet libraries between uh these various unrelated, like or semi-related uh releases, without having to coordinate like json bundler version bumping things like that, it's just kind of easier, but that comes with a a price tag which is, um if you want.

B

If you want a sort of minimal clean ci pipeline that doesn't run like hundreds of steps for your various projects, then um then you have to work at it, a bit like, even with even with git labs like nice ci system.

B

um So we uh we very quickly figured out that using gitlab ci's own yaml abstractions was maybe gonna, be in no offense to the verified, maybe going to be like a bit more, a bit more work than stepping back and building our own abstractions to serialize the gitlab ci yaml. Instead, and because this project was already json it heavy, we made gitlab ci dot, gitlabci.jsonnet.

B

Instead and before diving into, um can you see the terminal by the way on my other window? Have I been sharing the right thing? This whole time.

A

um Yeah, I see.

B

A

B

A

uh Gitlab cij signing up.

B

Yeah awesome: I was checking out she had the right screens the concerto, um rather than dive into that. Let's first look at how we want the pipeline to look so going back to this environments directory right, we've got pieces of software like evicted, pod, reaper, jaeger van ostern house, and each one of these has um the sort of pet name of um of one of our kubernetes clusters. So gprod is google production.

B

At least I assume the g stands for google rather than gitlab it does yeah just just checking that maybe it stood for gitlab the whole time- and I I forgot about that. But yeah we've got g stage g prod uh pre, which is a kind of like in for test environment. For anyone who doesn't know dev is the exception. Dev is actually just um a you know: relatively agnostic, sort of mini cubed k3d compatible uh release.

B

That's um that uh contributors reaper meant to try to use for iterating on something local before uh before pushing to a shared thing like pre or g stage, um and each each of um these environment directories contains what's called a tanker environment which is effectively a description of some like one or more evidence resources to deploy to our namespace in a cluster.

B

This video is not going to be about tanker itself, but I would really encourage people to check it out, because it's really awesome and I really like using it- it's instead going to be about how we coordinate ci to this.

B

So, ideally, when a developer pushes a branch we want to produce tk dips, which are essentially cube, cuttle diffs for the environments, they change and ideally nothing else.

B

When master is merged. We want to produce diffs again and then possibly manually gate the apply step, the cube cattle apply of those resources we actually at the moment. We do have that manual gating for prod environments, uh so gprod and our ops cluster, which is the kind of production, operational, observability control, plane, kubernetes, environment, where we run things like uh san os query.

B

But we we just auto, deploy staging and free and all the non-prod environments without um without any gating, uh because you know how.

A

We do it pretty much everywhere. Almost at this point exactly.

B

That's I realize that this is already in in the helm repos, but uh we we didn't have that in tanker we were clicking buttons in pre, uh so you know we didn't. We want to do that anymore, so we removed the gate yeah. There is some other nuance to uh tanker called pruning which is like apply, but for deletion, which is these days in line with apply. So we can think of this.

B

As um we've got a bunch of directories structured like this software named and environment names and when they're diffed by a commit, we want to dif only those environments and for master commits. We want to apply only those environments so that produces pipelines that look like.

B

This, um let me try and find something.

B

So if I grab this pipeline.

B

All right, that's a bad example. Actually we can maybe come back to that as a later example, because this is um this is another use case for dynamic pipelines.

B

Here's a master pipeline.

B

I'm looking for a non-master pipeline, so when I mentioned about changes, we'll come back to this, but often uh often for the diff stages. Very many jobs are run. um All of the jobs are run because of a gitlab ci documented quirk uh in which new branches uh don't have change rules applied to them. They end up triggering truth. Like every change is rule this. This is actually something I have an open issue for. The verify team are considering fixing, I think, which is good.

B

So I'm just looking for something that doesn't exhibit this bug pretty much. Everything does unfortunately,.

B

Yeah, we're not gonna be able to see the effect of nice changes rules, but we can see, for example, a master pipeline here um in which we have various kind of generic steps. We have one diff step per environment in this case. It's running them all. You have to take my word for it. Sometimes when we get really lucky, we do especially master pipelines, do see um uh only some of these running, but this is all of them in this case. So.

A

Just the ones that you're targeting so like, if I was only modifying this, this cuddle stuff you'd, only see this. This cuddle, like diffs.

B

A

Not the evicted, pod, reaper and jaeger and such absolutely.

B

We'll get we'll get like more detail exactly what triggers what we dig into the file um and then because this is a master pipeline. We have the apply phase.

B

uh We want manual gating for some environments like gprod and ops, but not all. um We want jobs to run as independently as possible. So, for example, we want the jaeger g stage apply to run when the jaeg jaeger g stage diff finishes without waiting for everything else in the diff stage. Using gitlab's directed acyclic graph feature needs keyword um and that's really all we need to see about our like main kind of pipeline.

B

So let's dive into the code and see how we've accomplished that.

B

So, starting from the bottom right, we call json it's manifest. Yamldoc function with a gitlab, ci conf object, um so this gitlab ci conf is some like. uh You know: json object that has all the various structures that make up a gitlab ci. It's got a stages section.

B

You can look at how this gets translated. Actually.

B

That transforms into this.

A

Yep which looks familiar, I recognize that absolutely.

B

uh Unfortunately, jsonic manifest yaml doesn't include any options for like pretty printing like removing quotes where it can more white space. Things like that indentation. So it's kind of a very, very verbose like standard demo, but yeah. We don't really look at the yammer very much exactly yeah. We even pre-pen this string at the top do not hand edit this explanation here.

B

um We have a workflow section this uh I might be able to remove this. This works around a um this works around a bug in github ci, in which, when a pipeline configuration reaches a certain complexity threshold, mr pipelines and commit pipelines, start being kind of arbitrarily double triggered, and you can use the workflow rules to pin one or the other, but I think this might have actually been fixed recently, so I might try removing this then going back down. We've got um some jobs.

B

You know, like you, know, formatting checking with the generated codes up to date. That kind of thing that's not particularly interesting. These jobs are just fairly vanilla. You know whatever yeah, then we've got these interesting parts here, we've got these tanker jobs functions being called passing in various things like uh like thanos g-stage. You know jaeger-pree some of these that have extra params.

B

So let's jump up and take a look of what this function does. Actually I want the I want to keep the yaml on the right, and so we can look at it later.

B

Let's jump to thanos g-stage diff in the yaml. First, apologies for the constant jumping. So what do we want to do right? We want to run some image that contains tanka. We want to I'll come back up to rules. You want to run some scripts. You know like log into the cuban environment, vendor stuff, blah blah blah run a diff.

B

You get it that's in a stage just for, like nice decoration on the pipeline, it has some tags because we need to access our vpc, so we pin it to certain nodes uh on from our op server.

B

uh It's got some variables, the utility of which might become apparent in a sec.

B

Yeah, like those.

A

Would help you log into the appropriate cluster that you're, targeting probably.

B

Yeah we can look at the cube control login script using some of these, so we grab the variable called environment, which here is g-stage and use that to reference the ci variable, which is a private key needed to authenticate the g cloud.

B

The tanker m's.

B

Variable value here is a directory path, environments, thanos g stage that we feed to tanker here.

B

To um tell you what to do. I mean there's more than one way to skin that cat. I mean to be perfectly honest. If I was writing this today, I would just.

B

uh Environment stan oscar stage and like not have it as an m4 because we're generating this config anyway. But you know whatever.

A

B

Yeah um and then we've got some rules. This is the most complex fit of the config all right. What is all this? This is more rules than we normally have in a ci step here at gitlab.

B

Well, first of all, we if you're familiar with the new rules, changes syntax as opposed to like the old, I think, deprecated now, potentially the only accept syntax.

B

We want to only run this job when one of these things has changed like at least one of these things has changed. So this isn't all environment standard g stage star. So if we actually change this environment and then we've got a bunch of like pessimistic constraints, um we want to err on the side of running jobs. So if we change the vendored json dependencies, the tanker uses that we can't necessarily tell which environment it's really targeting.

B

If we change the ci config run every job and if we change any of our shared like common libraries under live, because we unfortunately don't have a mechanism for um for, like reverse bubbling like oh, we changed this part of live and that's used here here and here we only need to run these jobs. That's way too complicated. We'd rather spend the money on on running for jobs. So we are on the side of running jobs, um so commits the change. Some core stuff will end up diffing everything.

B

But then, if you have a nice targeted commit that any changes, one environment it hopefully should should trigger just that job. uh And finally, the changes is added together with this. If, if the variable tank for deployment's run is set to one, this is an artifact of our mirroring setup.

B

We push commits to gitlab.com, um but because, uniquely unlike other customers of gitlab.com us on the gitlab infra team are like operating crucial components of gitlab.com, so we tend to operate that from another gitlab server, a private one, ops.getlab.net lots of people watching this video will know that we push mirror to ops.gitlab.net and on there.

B

Crucially, I'm going to try and not leak secrets, which I think is possible because um I think I shouldn't leak them, because this should be started out. Yep good here, we've got, I'm not going to click, reveal values, we've got tank deployments run and I'll. Tell you a secret. It's set to one right there and we don't. We.

A

Don't surprise, I'm surprised that you don't use the uh there's a referenced uh ci variable that lets you know, which instance you're on I'm surprised you don't just leverage that as a mechanism.

B

A

Url you mean uh ye, I think that's right like there's like an instance, url that says: hey, I'm ops.gitlab.net or I'm gitlab.com.

B

Yeah, it's. um I think we do that in the helm, repos uh yeah. That would also work just fine. I mean realistically we're very unlikely to change our server urls.

A

Devices, it serves the same purpose. You know yeah.

B

This felt somehow more explicit, like I'm, enabling tanker pipelines for this, but but no that works too and in practice it doesn't matter very much so this here will uh ensure that com pipelines are never triggered. In any case, dot com doesn't have the secrets necessary to log into the clusters.

B

So if you're, if you're watching this on the internet going to attack, gitlab you've got to attack the op server, not the dot, com, server um and uh yeah- that's that's the gist of it so because of the complex rules stanza because of the need to um control.

B

Like you know, I know I've replaced tanker end with this, but because of the need to kind of template a little bit of stuff, we very quickly realized that uh doing that with the amp with the abstractions provided in the gitlab ci yaml was maybe not going to be quite as nice as just simply serializing this.

B

So tanker jobs is a you know: meaty ashjasonic function, that's um starting at the bottom. Looking at what it returns, it returns uh an object with these jobs, so software here would be like thanos cluster would be g-stage, for example, put the stage in you put your oh.

B

You put your script in. You assemble your relatively complex rules. um The nice thing.

B

And this is this is another reason actually by the way that pushed um pushed me towards gitlabci.json instead of using like yamalanka's and yaml, includes because rules is an array, unlike the old only except syntax. There wasn't really a nice way to try this out somewhat and we would have had I mean there's about a thousand lines in here right. We would have had many hundreds of lines of repetition trying to duplicate the rules like um like this one and things like that. So it's a bit drier here.

B

You've got you know, rule enable deployments plus and have changed which, um what's the definition of them changed. Yeah right here, you've got your you know, because we call tanker jobs for everything, so we've got different enders for each then we add the kind of pessimistic part.

B

um Yeah we could, even here you can see we're using a json, it um extension so we're mixing in to this base job. We we, you know this is maybe a bit overboard, but like json, it makes it really easy. So, rather than even duplicate things like image tags and variables, we just extracted that out into an object and reference it from both the diff and the apply job down here.

A

B

um And that's really about it, because now now we've actually covered this whole file. Like it's, it's a meaty-ish. It's like 305 lines of json. There are no imports, though so there's all the context is here um yeah and it's so far generated about a thousand lines of ci yaml and we are nowhere near done, but just repo we're gonna bring in a lot more stuff, so we're gonna balloon. This config out potentially maybe hit some api maximum.

A

So we as people who are generating a ci configuration, should not touch the actual yamaha file that gets generated. What would be the appropriate command? Maybe.

B

A

Inside this tonka repo specifically, but what is the command that generates, that file.

B

A

B

You we'll look at the make file so make generate. Ci config um runs json dash, dash string, dot, gitlab ci json, redirecting to gitlab ci yaml.

A

B

And we have a ci step that um uh checks that that the generated code matches the final result. Now, of course, you know you're contributing to the ci config. You can just remove this chart right, but so it should work most of the time uh the string, so json is pretty simple. um The only reason for this is to say that I'm not json, it usually generates json.

B

So if you want to generate anything else like use, use functions like manifest yaml docs to return a string- and you just tell it don't- expect jason, I'm just going to give you a string as output, it's just some boilerplate you have to put on there.

A

Okay, okay, so from the perspective of someone who might be newer to json such as me, where would you go for the best documentation or learning tutorials that may be available out there? Two.

B

Places the json.

A

Documentation is uh maybe not the greatest, so if we haven't.

B

Improved a bit.

A

B

I would start with the jsonnic tutorial um on json.org uh this. This kind of has everything in it. It's done by example, which is nice, there's really not a huge amount to jace on it. It's um you know it's a smallish programming, language, similar to q or daal. If you've seen those that they're called kind of configuration languages these days, a niche for a niche for people who intentionally perhaps don't want all of the bells and whistles of a general purpose.

B

Programming language, like you, know, python um good for this kind of thing, so I read the tutorials: it's not super long, uh and then you've got the jsonnet standard library, the only and then beyond that, like once you practice with that a bit and looked at our existing json it, which um we use based on it in a few places in gitlab infra, we use it in tanker deployments. We use it quite liberally in the run books repository, which is actually a mono repo for way more than just run books.

B

um We use it as the basis of our metrics catalog, which generates prometheus configuration and dashboards for our various services um yeah. Beyond that, it's really just the context you're running json in like if you're running tanker, which this video has like broadly not been about like you've, seen me just calling out to json it here. um If you're using tanker, then technically you're you're, not like shelling out to a json executable. Your your json is consumed by an in in process json interpreter inside the tank executable, but that's usually a fairly minimal consideration.

A

Gotcha well excellent. um I don't have any further questions. Do you have anything else that you want to quickly gloss over.

B

uh No, I think that's about it from my part. I have a meeting to run off to in two minutes perfect, um so I think you.

A

Have to go well. Thank you much man. I appreciate the overview.

B

Don't worry it's up.

A

To youtube shortly so.

B

Yeah, I look forward to recoiling at the sound of my own voice.

A

All right, man have a good one. You say later on, bye.