GitLab Repeatable Database Creation, 20 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Repeatable DB creation Demo (2021-10-20)

Description

https://docs.google.com/document/d/1jGZow42nbmSOuK0TDv9WUHt9jkpUHDrt7acNL0R9DWs/edit#

A

A

In a minute for everyone to join so soon, who's around.

A

Okay, let's get started um again. I don't think we have anything to demo unless, if you guys have something um I'll start with all for the discussion, which is I'm working on this prototype to move ansible into config management, but before we do that, I want to see if we can move our enviable roles into a collection which I think will organize them a bit better and also we can move our plugins to a collection as well.

A

The challenge I'm having here is that what I would like to do is create the collection in the project instead of having a dedicated project for the collection I like to create, like just a subdirectory and this according to the docs, this should work, but I haven't gotten it to work yet um I think we're already kind of on the latest version, but um there have been a couple releases and I saw that there was a bug or two in um our current version related to this.

A

So I'm gonna try to see if I can get it to work, um that's pretty much! All I have um ahmed or alejandro. Do you have anything.

B

um Not really, I was gonna ask you related to the collections, because I know in a couple of occasions, we've talked about other projects converting.

A

Them to collections.

B

And uh yeah do have have, has the approach that we have evaluated for those other projects always been having a separate project for the collection? This is the first time we're trying the monorepo approach. I guess yeah.

A

I think so I don't know um grant if you're there. um I know that we we had considered doing this for get, but you know we decided to hold off on it. I think the proposal for get was to you create a collection within the get project right.

C

This is for packaging up and suppose, collection, yeah, yeah.

B

C

The plan is once we've got, we're still focusing on functionality right now to to help you and and horse, and once we've done that and we're getting there. 1.3 is going to be over the next week or so, and then we're going to be taking a pause to see. If we can now look at these bigger restructuring asks that we've had about the collection and stuff so um for the collection, um we're confident we can probably quickly move to make it actually in the collection format.

C

So it could be consumable, at least by a git url, um the actually moving it to be published to galaxy and stuff we're going to evaluate the time.

C

If it's really a big requirement, if people will need it, if people are wanting it which they have said in the past, I'm just going to double check and see what the temperature is on it and it would be a bigger deal for us to tackle because we we've overrated the precedence of variables and ansible for a whole bunch of good reasons at the time that we did it and the precedence and aspect was weird and uh um the default press.

C

This was causing us to have to write a lot of spaghetti code to kind of jump through hoops to see if the variables defined here and- and um it was annoying, but if people really wanted to be on galaxy, then, um okay, we'll look at we'll look at trying to we. We will review that, so it could take a little while, but we should be able to do a shorter win there to make it considerable.

A

Yeah, so I'm wondering um when you, um when you install a collection you can both, you can specify a repository and a subdirectory within the repository. So I'm wondering whether you'll keep the monorepo approach forget, which is I mean or whether will change the whole approach to get in general like right now, it's sort of like you vendor everything right you vendor get. uh You probably fork it first and then you start making changes and then you kind of have to update you know as get makes new releases. I guess you have to re-vendor everything.

A

What is your? What is what is your you know, strategy around this moving forward? Is that going to remain the same? Are you going to keep ansible and terraform in the same project?

A

um Well, if you create a collection, do you think it possibly will go into a separate project and then you'll uh import it like, or have you just not thought about this yet.

C

I really thought about it. Yeah I mean uh when you split stuff into several projects that it does come with complexity, um yeah and it kind of ruins the single source of truth that we have right now, which which has some natural benefits um nothing's off the table, though um terraform is probably more open to potentially breaking down into more modules, um but they would still be the same. The same project for ansible. um It probably is going to be one collection because it is kind of it is all connected.

A

Together, yeah.

C

um I mean the individual roles for each component could be, uh could potentially be broken apart in the future, but um yeah.

A

No, I don't think it's really worth it, but yeah yeah, I'm not suggesting we would have multiple collections because I, I think you're right, like everything's too coupled for that. But the question is whether you would create a dedicated project for just the collection.

B

A

I, and I honestly don't think that's necessarily a good idea, like I think you know at least with so one of the challenges we had with terraform was that when you have source for a repository, you can't specify a subdirectory within the repository at least the last time. I looked right, so this is why in infra we have like a project per terraform module, because um you know when you use a git repository as a source, uh it needs to be at the root of the project.

A

Now, with collections you you have the flexibility to specify like a source of a project in a subdirectory, so someone could like if they wanted to import, get ansible roles from the get project they could just specify the get repo and then the subdirectory to the roles or the subdirectory to the collection.

A

um But I guess I'm kind of more interested in seeing like how do you envision people using the collection if we do publish a collection for get, there are so many things that are tied together. It's everything is kind of coupled right, so um I mean we've really kind of taken. This monorepo approach, where you have your own ansible config and like everything, is very opinionated right. So I I don't know how peop someone would use get necessarily as a collection do you have.

C

An idea about that, so um everything everything is connected uh coupled loosely, I would say: there's no, I mean once you go into ansible stuff, that's where it starts to get a bit more uh more connected, but I mean it is possible today uh to break apart from an aspirin into several projects that that would not be an issue um that that is, that is doable for terraform. um It's a different release, process of terraform. Obviously it's not the same as collections. Terraform modules are actually supported in gitlab.

C

That's probably uh what would be what we would look to do for terraform, so it's a slightly different process um for that. So I don't know if that immediately is something that we need to like look at breaking up the projects to to support. So um it's something I'm always thinking about. I mean up so for now to to to handle the capacity that we have running get is is to run at a source. That is, that is the design.

C

For now, um once we get the functionality out that we need to get out, then it started to go back to the drum board, okay and we're looking at going. Okay. What what is now the recommended way we're going to go forward with to to run get um and that will inform where we will go next, whether it's collections and terraform uh different registry or ansible galaxy collections, um it's up in the air, um because each of those they come with their pros and cons. um I need to touch base with cristiano people's house.

C

You have some troubles running terraform tests when stuff was more broken apart among his collections, he was actually sometimes some troubles there, because terraform.

A

C

To get quite terrifying can be, terraform is generally very good, but it can get itself confused, sometimes about cycles and and not being able to derive elements at runtime, uh well, pre-compile, really using the wrong terms. But when you start to run terraform it usually can try and figure stuff out, but sometimes it can and then it gets itself into a whole state when breaking power and using stuff collections. It seems to do that more so the stuff I need to look into, but we're looking.

C

We always look to release it as a docker image as well, but it'd just be the source in a docker image, just for easier sake. Before.

B

C

Available but um I'm happy to to take advice and take prefaces from people to see how they want to run it.

A

Yeah, I don't know whether it would make sense to publish a whole bunch of terraform modules and release them independently of each other. I could see like um I could see us publishing all of the ansible roles together because well, I well yeah, while I think like tariff you're right, terraform and ansible are very loosely coupled really it's just the the labels right, like you know, like.

B

Different machine.

A

Types and that's it right so so I think, like you, could release terraform and ansible separately, but I, but I don't think you can take a step further and like you could, but I don't. I wouldn't recommend releasing each ansible role separately, so you would want to use a collection for all of your ansible roles and then for terraform modules. I would say the same like I don't think like you'd want to create a project per in a per terraform module.

A

I'd probably want to release all of the terraform modules together, which case I would say why don't we just put everything together in one project like we have it now. You know like I, I I don't really see any better way. I.

C

Need to dive into the terraform functionality. I think I believe it's possible to publish more than one module in a project, but I could be wrong, but yeah I mean the modules terraform. The reason to mention it. That way is because your cells are using only the instance modules and not the reference modules which and the reference modules needs- are completely dependent on instance, modules, but that would be.

C

Yeah, I completely agree, I don't see much of or any need really to individually publish the roles. They have been the additional work there. Just doesn't isn't going to be worth it. You know for collection, which is just text, it's just easier to have the whole thing so and because the roles are are all dependent on shared variables. At the moment that will reduce the future, but they will just be some variables that are always going to be shared.

C

That there's no reason why they wouldn't be so um they're also going to be broken apart either. um It's just it really will just be an exercise of what is the need, the only the only definite use case I've seen it is that someone wants to consume the ansible rules as a collection via get url, and then everyone's then followed up by saying. Oh, it would be nice to publish in galaxy, so it's just kind of going just picking it down a little bit more and saying I.

B

C

Want to a galaxy or not it's the question right, so I.

A

Don't think it's that, like galaxy, doesn't really matter, I think they, you might.

B

A

Use a good url you know like or for the first iteration of this, like there's no sense in publishing the galaxy like. Doesn't I don't see any advantage and- and I would even like, take a step back and kind of work backwards from how people are going to be using get like if we're thinking just in terms of get like right now? um How? How are people using get, and I think how they're using it right is their forking the project and then modifying it right.

C

um They are, there were some forks in the beginning and that got a little bit reduced, um but they are basically taking get and then they're using it as the master essentially and they're doing stuff around it. So what I think I've seen so far is that usually you'll see wrapper scripts or some other tools being brought into, because you don't get only do so much, there's other stuff around get. That would be more. It's complete, depending on the project yeah.

B

C

um Usually, what the the the only things I've seen that have actually are tangible is people just taking get and using it as the kernel of their of their work, but then building stuff around it, like horses, using dead, cuttle or yeah.

A

Yeah but then, but then how do you update from upstream, if you do that like how? How is it like, I think horse is doing like a vendoring approach here where they're just taking, but I don't think that's what we want. Everyone like what.

B

A

What is what what do we want to if we could recommend one way to use get um my my idea would be would be like. Maybe you have a way to bootstrap um a scaffolding of a project where we reference get as terraform module source using git and ansible collection source using a get path and, like maybe there's even a script, that would create a skeleton of a project uh in gitlab. You know some scaffolding that would set things up kind of for you and then and then you just update to get upstream changes.

A

You would just update the sha or the tag for the collection or the terraform.

C

Yeah I mean that's speechable for sure I mean the only only.

B

C

Is for ansible, um you will need to download dance for config file that we have right now too yeah.

A

Right, yeah, which.

C

Is annoying, I know it's my biggest issue of ansible, except for that I'm quite happy if it's just depressed, this is just not good for how you use it, but the um they get they get. Urls is probably the more reachable thing: it's probably what we'll do first and then it will be a big ex exercise of this anyone that will really require anything else or is it just or people just kind of spitballing saying: oh well, you know so the um for it's a weird one, because you know these.

C

We from the beginning have been very boring here and just said we're going to be using pure our antibond terraform and that's by design, because why? Why would you try and recreate or do anything else here that these.

B

C

Are very sturdy and very mature already do and that it that alone then meets it, informs how you run it. Anybody even ansible already have answers to the machine. They will be running from source technically, because they'll have ansible config files and terraform config files on their machine and whether they again either refer to the ansible terraform stuff, whether get url or if they just get checked out and just actually refer to it locally and disk. At that point, it's it's a little bit of semantics.

C

Obviously ci is a little bit different, but at that point it you're pretty much following the same process is there is there need to then publish your own get and galaxy and telephone registry or or anything else? I agree with you, probably not I I I don't see the need for it for the extra maintenance that we need to do for a little game.

C

Maybe we are informed by the tools themselves, so um uh we'll probably we'll look at the get piece to get url piece to begin with, and then we'll probably look at that if anything else is really required. After that.

A

Yeah, like kind of going back to this idea of creating the scaffolding for a new project, you could probably do this with a project template that would include the ansible config, but not much else, right, amazable, config kind of like the basic, um the basics. You need to get started, but would mostly reference just um the get like get repository path for terraform and the collection, um so that it'd be easy to up to to take new upstream changes. I think that's the challenge right, like yeah.

C

We have now yeah, I mean obviously changes to answer that, particularly right now.

C

What you do is that in your ci or in your scripts, you just have you know you you'd likely be pinning get to a tag, and I get checkout and you just be updating a tag manually, probably after doing or as part of a process, to make sure that your the latest version is all good for you, and then you know that would be taking a step forward, get urls as well, uh where you probably would want to put your in your get urls on a tag as well too, to prevent any big changes coming in or you know to prevent any uh issues.

C

I guess um the uh yeah um it's interesting. The project template is a weird one, because for this particular tool is that's a cache 22.. You don't have a gitlab instance that you sign up get for the first time. So what do.

B

You think how do.

C

You define the template, do you find it.com.

B

That'd be weird.

C

If, because at any point, you'd be using get to be building your own instance, so um we could do like a script or something maybe but then to get a script you need to check it get get. So again it comes back to source. That's why I've just.

A

C

Sticking with source, because it just makes a lot of sense but happy to explore other options. Yeah.

A

Oh, you mean like stick with source, as in you would download a tarball at the project to get started. Oh.

C

Yeah good check out orator for now yeah.

A

C

Get urls is completely fine and we want. We definitely want to support that um for the next next release for sure. um But um as I say I mean it's, it is just by the nature of the tools you're you already are.

C

The opinionation is already there not from us, but from asphalt and terraform you're gonna have to stall on terraform you're gonna have to start answering your ci or create a dock image that has the military installed or something along those lines, and then you're going to have files on disk that will inform both whether they then refer to each of the modules and the roles as urls or local disk pass. Like say at that point, it's just a single link, saying what? Where do you go?

C

What do you do um so it we're just kind of you know a sensible terraform really.

A

Yeah yeah, no, I see I see what you're saying I just like. I I still think there's there is a use case there for using. There is a use case for using get to deploy a gitlab instance from gitlab.com.

C

This is a use case.

A

Yeah, if not like- maybe not so much in the community, but definitely.

C

A

I think right, like this is yeah something, so you could see how a project template might be useful for that, but yeah you're, probably right like, but.

C

Then why would you add the template purely for devs to use then, and no one, no customers, we use templates for customer facing. So it's all.

A

These questions.

C

Keep popping up you see, so I always I keep you start to go down these paths and then you keep getting buffered back into oh we'll. Just let people check out. That's easiest.

C

I mean it's also the target market as well. The people who are running this would typically be. You know, experienced individuals who are used to terraform as well. They likely already have them both installed. They probably are used to going through and doing these kind of things themselves, um so yeah get urls is definitely a path um after that it's uh well.

C

I I don't know. If there's going to be a better deployment method, that people would actually use outside of using get your rails of source.

A

um Yeah, I guess I'm still missing something here, which is how people use get, because if you just download the source, you're gonna have to update variables in the get repo right, you're gonna be making changes, you're not just going to be taking get, as is you're going to be modifying things and you're going to be committing those changes and now you've diverged from upstream.

A

So would you just rebase your changes on top of the latest get tag or.

C

You know I mean that's how, when we did the first release of get way back when uh the first that was the first challenge I had to tackle, so we needed to move the bits that are config, that are user config.

B

C

Get so get there only contains the actual executable files, but not the config. So to equate to gitlab. You don't have a gitlab rb in your in your uh in your gitlab checkout. Obviously there's a default, but you wouldn't have a custom gitlab rb, which someone would use you go in and you actually create the config files. You need manually right now and we've left it open. We do call it the docs where at terraform you need obviously the main tf and a fewer.

C

We call it two other files that we just you've defined, that you define the variables that you need and you define the call to the actual module you run and then, first of all, you need to run the you need to run. You need to set up the inventory file which again would be custom. I would always be different per customer and some some and then the variables you're passing for that inventory.

C

Just customize, say passwords and stuff like that, um so that's already pulled out and that so for every every person running it they need to obviously find those files just like they would any other software. You need to define.

A

Yeah, where, where were those files like those configuration files, have to be checked in somewhere? So um is the idea to have get as like a sub module and then like, like what repository? Would it be checked into so this.

C

Is the part where.

A

I'm confused yeah.

C

No, so um so what how we do it and what we leave it open, because it's completely up to the user, how they they can literally just have the file somewhere else and just copy them in to the folders. um There's, there's flexibility as to where you can have your config as well.

A

Like okay, but I need to check it to get right like I need. Yes,.

C

I need to have source.

A

Control, so what would I do if I, if I want to run, get and I want to make customizations to variables, um I need to check that in right and then and then I'm diverged from upstream.

C

So for for this, what you do- and this is what we do in quality now- is that get doesn't get changed when it comes to these variable changes. We have a separate repo that just contains the config files for every environment that we deploy and configure and then at runtime during ci. We just have a simple script that takes those files and puts them into the places we want them to be, and then we can run them. So we check get and copy the files in.

A

Yeah so so I guess the procedure then would be like, for example, the environments folder under ansible is like a place where you would copy and files yeah.

C

After you, purely as an optional folder, you can that we just we just give guidance of here's a place. You could put your config files if you still wished yeah.

A

C

A

But you would check those changes in and then I guess, if you needed to take the latest version of get or a new tag of get you would you have to rebase right, you would have to rebase your changes um on top of the whatever version of get so.

C

You're updating for those folders we offer them as optional places for people to put stuff in. We expressly have configured get not to take anything in them, because we want to prevent people from committing sensitive data into their repo.

A

So you have those get ignored, we haven't gotten ignored.

C

B

C

So for our docs, the way that we visualize the people running get right now for customers is that they're, just checking out get and adding the files on on disk and then they're just running it. um They would then need to configure where to put those config files in a different place. Again it's up to them. How do you want to do it? Yeah, basically get doesn't care and get doesn't advise really how you do this as long as the config file, it's like gitlab, it's like omnibus, get lab.

C

It's like it's omnibus will run happily as long as the the gitlab rb file is in etc. Is there and configured doesn't know where it comes from, doesn't care where people store it in their world if they have a backup.

A

Yeah, I know I I know I understand that model, but this is much different because you don't check, get lab rb into source control like.

B

A

Not like, whereas other things you know like ansible config, you would in terraform config you would so. I think like. Like I don't know I I honestly don't know how other people are doing it outside of horse, but um I'd be curious. Like.

C

How they're working around this well! Thank you. What we've done! What we've done is that if in the zoom chat there there's a link to the project, we have for all of our environments that we test against. We have a whole bunch of them a lot, some of them weren't deployed but they're there. If you need them- and we just have all these different folders and they both contain the asp on the terraform config, and then we just have a simple javascript that we run during ci. There.

B

C

Take all these folders and copy them in and then run the one that we want. Yeah.

A

C

On my local desk, I just have some links so yeah.

B

C

I am not aware of any other better way to do it really um because, like say these files usually have passwords in them and other ssh keys and various other things, so um they can't be. They should never be committed for an actual main get repo um they should be. You should be kind of match them up as you need to, depending on yeah.

A

I guess I I think, like kind of the idea, that's probably worth exploring, is you know whether we could just use get as a terraform source as a git repository for a terraform source, for like kind of like a master module similar to how you have the reference architecture and then and then an ansible collection as a git source? And then then you have a very like small amount of scaffolding for creating a get project that only has probably just the ansible config and some other like small configs.

A

That can use the get repo as the source for terraform and ansible, making it a lot easier. If someone wants to commit their config to git, it makes it easier to update from upstream.

C

Yeah, I mean that's and that's definitely what we're going to be doing yeah. It is a streamline of the process. Basically, you still have our separate people filled your config, but you'd have them referring to stuff, not globally, but externally you'd be having them point to urls so and that's doable um for sure um the yeah for terraform. That's we just need to publish the module somewhere and for ansible yeah. We do.

C

We need to restructure a little bit to publishers uh to have it structured as a collection but yeah, and that should be doable once we get those changes in so then you wouldn't need to do any checkouts and copies you just they would just refer to the repos externally.

C

That would be doable. Yeah cool.

B

C

That's probably, I think, in my opinion, I think that's probably the way that most people run it. um I think I don't think any other process would be better than that.

C

You will need to have your files, you, your your behold, intensible and terraform's design structure, as well, so um without having to check it, get and copy stuff in taking that away is good and then, if you just have your files here, only the files that you only need to configure how you want the environment to run and then they pull and obviously get stuff from the internet.

C

uh I think I think that's probably gonna be the ideal. The way for the foreseeable, but again I'm happy to look at what people's preferences are. If there's any justifiable use cases for different ones, sure but yeah.

A

Cool okay, um sorry, I'm gonna know andre. We kind of got off track.

B

And I was ask- and I don't know sorry if this was made clear before, but how are third parties using get like? Are they deploying their self hosted instance, or are they using it to create reference architectures or what.

C

uh How horse using it, how.

B

Like yeah, how are third-party people like people outside of gate lab? How are they using get? What.

C

Are they good, for my expectation, is for most actual customers and users who aren't doing like a constant? You know automated process to deploy multiple environments or like we are doing. Our use case is a little different that they only deploy in one environment so for those users they're just checking and get and setting up the files locally in their on their system.

C

That's probably how most individual it people or or essays are doing it on on their machines. Obviously, when you've got team, that makes a little bit different and that's where you need to explore where to store config in a shared manner, but um you know I I expect people to be going down the most simple route of just doing on disk, but um I'm always keeping an eye.

C

I mean customer usage is still quite low and from what I've seen uh you know, that's individuals doing it and that's how I spent all work for most, but um I'm always keeping an eye to see if other people do it differently.

A

Well, um I think we are pretty much out of time um ahmad. Just on your side, I know that you have the mr to omnibus for the gitlab exporter console service. Did you um oh okay, so it looks like this has already went through a round of review, so basically we're just waiting for that right.

B

Correct yes, yeah, okay, cool.

A

All right, um that's it thanks! Everyone uh see you um probably next week and next week will likely be my last week before I go on parental leave, ciao.