GitLab GDK and Gitpod walkthroughs, 28 Feb 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: GDK::ConfigSettings deep-dive

Description

In this recording Toon Claes will take a deep dive into GDK::ConfigSettings, the configuration system used by GDK.

This is a very rough first iteration, made with zero preparation. I will make a second version (hopefully a bit shorter), but I'm posting this one already for early feedback.

Issue: https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues/1284

A

Hello there today I'm gonna record a deep dive into the conflict system that we're using inside kid inside GDK inside the gitlab development kit. So let me browse my tree here and let me show you or what we're at so.

A

We are at lib and we have GDK where we have. Basically, the the main class of all things is the config settings clause. So this is the class that we can, that defines a DSL uh uh to create a config file like the one we have in GDK.

A

So, as you can see, so we create a config which derives from the config settings clause on the left and we specify a large number of settings here and these settings are nestable attributes inside the class and we can um yeah. We can specify them as we go and have different types of them that but we'll dive deeper into that. uh As as we proceed so the config file, as you can see, it's, um it specifies the type, the name of the attribute and a default value.

A

You can make the config settings you can make each setting to use like um code Ruby code to derive from other variables or to run like here. It's it's locating Checkmate from uh from the dev, uh so you have here, config Dev, config Dev is over here then double underscore bins and that's the goal part that's been so it it finds it in the go board.

A

Where check make is that's where the binary is um I suggest we first start with like a generic string type for example. So we have. uh Let me find a really really easy value that we can use.

A

um Okay, I, don't know by heart, but anyhow, so we have different types here we have bulls have been teachers, we have arrays, we have um strings and stuff like that, and basically we, um these DSL methods are defined in this here. So we they're just like syntactic sugar around how you can find different types of um of config types. So it's not really important here but, as you can see, there's like a config type class for each type of attribute.

A

We have then, uh and that calls the def attribute methods which is doing a bunch of other stuff, but we'll dive deeper into this. So all these are- and that's that's, the main thing main takeaway. You should have from this presentation.

A

These methods are closed method methods, obviously so they are created at um when when the Clause is loaded into the Ruby memory. So when, when Ruby loads, this um lose his claws. It's evaluates these Clause methods to Define the attributes and stuff like that. But that's that's. That's that's obvious, but the block we're passing to those um class methods are actually instance evaluated and um like, for example, we have I, don't there's okay.

A

So, for example, we have this settings, so we have like GDK settings and that's in there are experimental settings and one of those settings is quiet. So quiet evaluates to config, GDK, quiet and yeah. We we use a question mark to make it a predicate method, but that's not really important. So this method calls config GDK client, which is actually just this method and just returns true. So this is the way that we um get rid of like or not really get rid.

A

We we keep supporting the uh the experimental method, quiet, but just depend on what the value is from uh the non-experimental quiet space. Basically, but in this block, config is an instance of this Clause of like the whole clause in this file, and we call GDK on that and we call quiet on that. So basically, it's the same as if, if you would do GDK config get GDK dot, quiet, it's true, so we get. We get exactly the same thing here and you even could do gdk.x period mental.

A

Let me see, if our spell it correctly: no I I'm, not experimental.

A

A

So this is like a yaml representation of this sub config settings clause, um so this is. This is really convenient for debugging and stuff like that, because it uh yeah you can see what's happening under the hood. Okay, I think this should work as well. Okay, this works as well. So when you define a Boolean attribute, we create the method for both like the normal name and then and the name with the question mark appended to it.

A

Just just some syntactic sugar again, um like I, mentioned it's important to know that that this block is evaluated at um instance level, and this like and and the definition of the attribute is evaluated at class level, and this can be really annoying and confusing confusing because yeah, there are different stages of the um all the stuff happening now I'm wondering if I can easily demonstrate.

A

What this means, because.

A

Well, maybe if I would um config so if I would say, I think it's like anything, and that is an objects.new.

A

Is this the way you can do it so no yeah, it returns an object, but the funny thing about this and now I have to do. uh Let me quickly load uh Ruby now I have to do I, think IRB. Okay, this is the command, so this this is load slip into the into the include slip into the load, part and um yeah. Something like that and requires GDK. That's the thing so now I can say: config is um G, decayed colon, colon config, dot new right.

A

So now we have a config and we can call the same thing on this. So this is my object is created, um but, as you can see, every time I execute this function. This method, the address of the object returns, is different. That's because at the moment this block is evaluated each time we need the value, so that's a little bit annoying I would say- um or at least for some people it would be annoying, but you could- uh and that's the funny thing about it like.

A

If you would say, um is the law I, don't know, I'm, not sure if this works now this doesn't. This does not work, I. Think if I new. So now, if I do config tone again, then I get the value from the from the yaml.

A

um So the the yaml is loaded when you create the object, when you create the config object, and now it just keeps reading uh the value from that yaml file.

A

um Maybe I'm skipping ahead a little bit of myself here because, like parsing, the ammo uh is something you will dig into a little bit later, but anyhow. So these are the helper methods that um that Define, a DSL actually to um you, have to to have to make it possible to create attributes or to to specify attributes for for your config and the magic starts over here with the dev attribute um method, so it shouldn't be called directly.

A

It should be called by one of these helper methods and it takes the key which is like the key that you have in your in your DSL. So basically quiet would be the key for for this one and it takes a clause. What's the type of that, that's specified that's selected by the helper method around this thing and then you pass on whatever the arguments are and what the block is. So that's important to remember so now we pass on the Block to that class method, and over here the block is just returned.

A

True, basically, uh we we don't use any keyword arguments. So let's ignore that for now then, for this, then, um on the clause, we specify the key in the attributes we we put in the key and we put in a config type, Builder object. So we don't really create the method. The method on well hold on a sec.

A

We don't really um evaluate the block at this point as you've noticed in my demonstration a little bit back, but we assign a builder object to the attribute and what it does mean under the hood is that we Define a method for the key, but we built we use that Builder object to build the.

A

Yeah not really the block basically like we built I'm, not sure how to call it um Dev build. Let me let me see how yeah so we get the attribute we get the attribute for the key and we build it with self as the parent, because we need, uh as I've demonstrated earlier. We need to make it possible to access the config from from the methods From the Block in the um in the build in the Builder. Basically, so we have a builder and we build it and that Builder returns well.

A

What does the Builder actually return? That's a great question anyhow, so we can go to the config type and, let's see what's in the Builder, that would be important.

A

So return we return an instant of clause and that's the thing right. We return a config type depending on what the um what was selected from the DSL.

A

So we built a builder. uh uh We assign a builder to an attribute in a map. Then when we call the methods we look up the Builder and that Builder will create a country type of a correct type and we can get the value from that.

A

So that's basically what's happening in the Builder, so it's hold holds on to all the information um to X to access the attributes when we need them. So we we know what the key is we know about. The Clause is so that's the conflict, type argument, the keyword, arguments and the block. Basically the block that's passed on to uh over here. So the true. This block is kept in this in this Builder, so that's kept at class level.

A

So the Clause um knows what the attributes are so I think config does not have any attributes well I. It might be that we um delegate attributes to yeah with Elevate attributes to the claw. So what we basically do is uh GDK config would return the same thing right um so now we have a clause so when, when when, when the config file, so when this file is evaluated, we we built the class we built how the Clause, how the all the config um is evaluated, but we basically don't do any. We don't create object.

A

The conflict object yet um so there's no um Singleton object of this of this config. When we, when we evaluate this class it just it knows how to build a config object, it doesn't really build it. um Is there anything else? I need to tell about this Builder yeah, okay! This is something else that we um that we will might look into later now.

A

So we have a bunch of config types and I think we should have be looking at a base type first, so I think all of them derive from the base type. If I'm not mistaken, maybe the settings doesn't but uh we'll get into that a little bit a little bit later, and um so, as we've seen in the Builder, when we built an object, we asked the cloud we initialize the Clause, so we create an instance of the class. So that's what basically, what we're doing over here um we we set. Who is the parent?

A

That's the parents uh config object, because we can Nest the countries in each other more on that still later, and we say who is who is the Builder um for the value um as as I've briefly touched on, so we have, uh we have a gdk.dml, so you can specify a bunch of settings in this file. So if you want to use your own values, you should specify them in this yaml file and I am pretty sure that this method fetches the user value from the parent yaml. So because again attributes can be nested.

A

You have to look for the value in the correct tree item and you ask just your parent: has a yaml, which I've also demonstrated like a little bit earlier, when I saw that, like there's a subset of the ammo, um so we can get the user value from that and we store it in our uh settings.

A

So this is the value and this is their um yeah. This is our event, so that happens and on initialization, so I'm, not sure I think if we would say config build.

A

uh Yeah, that's what I was afraid about. Cent builds.

A

Does this work could not fetch attributes for tone.

A

ah All right, so this is what build returns so the Builder this object and, as you can see again so this normal I suppose this object, ID will change, but this won't uh the Builder won't because we store the Builder object in the class and the class won't change.

A

So if I call this again, this address is different, but this one ain't, um no I'm lost my trains of thought a little bit, but um so that that's what the Builder, the Builder returns a config type object, and generally the user doesn't use this um comfy type option. It will it's only interested in its value and that's what you get when you call the.

A

um Called the the named method by the attribute name, you you call Value on on the config type.

A

So that's what's basically happening under the hood, but there's there's more there's a lot more happening, because um what is what is the Value method actually doing.

A

um There are a few things to keep in mind, but let me check uh what's Happening Here, ah okay, I wasn't sure about that, but it seems were we should be, should only be.

A

huh We only should be evaluating the.

A

The value once.

A

Okay; okay, that makes sense, because.

A

I've demonstrated that the the object we return- um okay now I, have to. Let me refer to my ammo again and and okay I did this new than this uh yeah. So this return is not new object. Every time. That's not because we.

A

Evaluated the uh um the block every time, because we don't store it, that's because the Builder creates a comfy type every time again. So if we would say uh this is config uh hold on a sec, let me copy this because I'm too lazy.

A

So now we store the config type and if we say tone dot value, we get an address, but we always get the same. Address yeah!

A

That's because like when the object is initialized, we read the value and we we use restore that value in an attribute in in like a basic Ruby attribute, it depends on if we have a yaml attribute defined or not. If we don't have a yaml attribute defined, then we use a default value and a default value, and here is like one of the most tricky Parts I. Think in this or well. No, not not the most splitted part, but one of the tricky Parts here is like on the parents, which is a config settings object.

A

We instance evaluate the block and that's the important thing here, because it's instance evaluated and that's what I've mentioned before as well, because we evaluated on the on the instance and not on the class, and then we call the Block in this case. It's just a simple true, so it doesn't do anything special, but it could be calling config and or other arbitrary Ruby code.

A

um What else do I need to tell about this.

A

Like there is a validate in here, so there's a validate method and in this case it returns true. But hmm why ah right so this this construct looks a little bit weird. Why aren't we doing this? And that's basically because we want to call this Setter method and parse the value and parse under the hood? Where is it where's my parse huh where's, my parts?

A

Okay power should be implemented by the child classes. So if we say anything, parse just returns a value, but if you say string, then we say: okay, if it's nil, that's nil value, uh then it's a type error. But if it's not nil, then we return We, Run 2s on it. So what this does is because uh um so tone is my.

A

So if you I would say tone, dot value is one which is is number internal value.

A

This ain't right, this isn't anything of obviously, okay, this isn't anything thing. um I was not using a string. Do I have a string at hand somewhere.

A

Yeah hostname, okay, hostname, so config hostname, that's a string, I could say: config sends build. Where is it um hostname?

A

A

huh Is that ho H dots builds ah I, don't.

A

Builder value, all right so H I can say value is one but if I say value, it's converter string. So that's what the parsing does under the hood. It sanitizes your input, basically uh to fits fit. The type that you need.

A

So that's that for string like that's pretty straightforward for integer is basically the same, but as it does the other way around it's it's it um checks if you can convert it to uh to a number and it checks if the value actually is the same as you wanted. So.

A

um I guess comfy.

A

Ps, no, not that ort, so P dot value is that value. Okay, but I could say p. Dot value is one two three four and then period value is also sanitized to a number. But if I would do it like this, like a extremely high number that we are not able to part into the teacher now it says: okay, it's signed, but huh did it work? It did work. Okay, maybe my number still needs to be higher.

A

Interesting I did not know that we could store such large numbers.

A

But okay! Well, why not to the easy example.

A

Oh, is it for both like 4.0, ah okay yeah, but that's the cause is not an integer yeah. That's true, like floats, are obviously not like teachers.

A

huh Okay, then this example does not match up I'm surprised that we can that Ruby does this I did not know that anyhow, that's uh that's beyond uh this demonstration. So what else is happening here?

A

Airport I'm, not sure if Port does anything else?

A

Oh yeah, okay, there are special things around sport that are not relevant for this.

A

um Let me go back to my conflicts, headaches because that's a basic log base class. So we have this DSL methods. Then we have like how it parses the things and then we have a validate method and it erates to the whole tree of attributes to validate all uh all values so that when a user specifies a value in the yaml, uh it's actually a valid a valid yaml and and the user is informed about about errors. As soon as we can uh dump is basically a method.

A

That's called um when we also like do the GDK config cat command, um there's some stuff going on because, like we don't and that's one thing to keep in mind for config, you have these underscore underscore methods, so it basically you have this underscore underscore platform, and these are um like hidden methods, so they are not dumbed, so I think uh CD level gets a GDK, config gets and then I say it. uh Blood. Well, I think this works. This works, but wait. Let me.

A

Let me take another example:.

A

Like def check make Dev check make, so we only see the version. These two attributes are not shown because they're basically hidden with the double underscore, and that is what this function does. This matter does, and we see this ignore function. We have seen in the I wasn't in the Builder yeah over here.

A

So that's that.

A

um Command yeah: this is, it allows you to run a command inside the inside. The block of a config setting same for read, executable same for a finance Google same for read, then we have usually defined which calls on the base Clause calls user defined.

A

So it checks if the user did Define a value in the ammo and if not, then we we just Define like okay, it's the default. Fetch is called under the hood to find an attribute, and things like that dick is also fine, is used by config setting. So if you do this, then it says Dev, and it will dig into this into this part. Basically in inside the the ammo.

A

um So these are like just convenience matters bury is like the opposite. You can actually bury values um again in the config. So uh comfy confident tone is this object, but I could say uh confident very.

A

Tone uh is it like this: it's like a new value. Okay, so burtito high can't face is high, so you bury the value. But it's not that's it's only in memory. It's only.

A

um Pertained in probably in a build, no nothing yeah, probably in the Builder or not no, it's in the ammo. Obviously in the yaml I'm, not I, don't think you can unbury uh value. So if I say config Dot yaml, then you see this value is basically written here.

A

That's very safe, yes allows you to save the ammo uh there's some stuff yeah. So slugs are basically these things that I've been naming like this. This is like a slog for one setting.

A

It's it's basically, the whole part that leads to the the setting in the tree of settings inspect is just for convenience to say, method, missing.

A

um Not missing I'm surprised, we still have it here.

A

Provide a shorter form for conflict settings all right. So if you have a predicate method um and that's the enabled method.

A

um Then it calls that on that.

A

So on over here you have config.ng Nicks, for example, uh can I say, dump yeah, so it has a binary and it has an enabled attribute. So I could say the config nginx dot, enabled question mark, but I could always also say this and that's basically a synonym for that again. Some syntactic cure um yeah board manager.

A

um That's something we added quite recently, so um GDK keeps track of which ports are in use um by by all the services there. So I can say: config.port manager.

A

Wow, this ain't right, huh interesting, probably because uh yeah but anyhow I, could say to the port manager and I'm, not sure, if that's possible, like if I say, config dots, engine x, dot.

A

Port can I assign this I'm, not sure uh 443. That would be really cool.

A

Yeah that that won't work I would need to bury.

A

um Engine X port.

A

Like this, huh why is there no nginx port.

A

Is this no nginx port, all right, it doesn't use.

A

uh Okay, this is not. This doesn't seem right.

A

So if I would say bury.

A

Course, uh configured port ah okay. This works perfectly so you can assign the other different services to the same port.

A

Cool cool cool cool. What else do we need to check out attributes some um enabled value load channel? So that's that's the short gist of it.

A

um I hope this is clear. I, don't know what else I should be explaining at this point.

A

Well, let me know if you more have more questions thanks.