GitLab Distribution Team Demos, 29 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Orchestrator 2020-05-29

Description

GitLab Orchestrator: Running on latest from Geo Playbook Merge Request branch https://gitlab.com/gitlab-org/gitlab-orchestrator/-/merge_requests/8

A

Hello, everyone and welcome to the Friday May 29th demo for distribution team. My name is Robert Marshall I'm going to be demoing the current state of the orchestrator so on the plate. Today we're going to set up two clusters: one is to have an application. Node three council nodes to database notes, and that will be a primary cluster named main as seen here in this description and the secondary cluster named go1 is gonna. Have it with one database, one application node down.

A

This is gonna push through, and provision in terraform on GCP me go ahead and get this started, get the first take a little bit of time.

A

So it's going to provision to terraform, laying it out lay showing this down here. This bottom left corner of my screen, so this is a description file. This is what's going on when we're doing geo, we added config set set up yeah. We have cluster descriptions which describes the cluster and then we're in add geo, which isn't says our enablement and then names which of these define here, which of these clusters is our primary cluster and then also defines a listing of our secondaries.

A

Making. This more explicit helps us in the cases where you maybe have multiple clusters, because this bull is saw multiple. So if I want to say install, you know, tus Custer's, for two teams now in an NGO cluster for the main org I can do that. Have three standalone clusters, one of them having a Geo setup. So the configuration today is much smaller to keep it from running, for it keep the demo from running a long time now.

A

The affected outcome of what we're at right now is that we are going to get a primary up and running, and it's going to have the Geo turned on and enable and we're going to get all big secrets, SSH keys copied over and all of that is going to be there.

A

What I'm expecting is that one of two things is going to happen on the secondary one, there's going to be a global database, migrations Pistor. When I was testing yesterday, there was a a glitch where it didn't set up correctly because they didn't go in the right order. I fixed that manually and I fixed the order in the in the playbook.

A

So this will be my first time running it with all the fixes applied all in line and run at once, like I added everything, I think along the way to the play books, but this will be the first end end with all those fixes to play I've also fixed things. This adds there's a SSH, fast keys, so that's gonna get installed today and we managed by Orchestrator and.

A

That's where that's where this is going. While this is running something we can do, hmm I'm gonna think this is a lot bigger.

A

So this is just a sample of what the configs look like when they come out. This is from my cluster from last night, so I tore this down, so all these IP addresses are safe to have out there because they no longer exist and all the passwords are ephemeral and they're protesting only so this is the setup you can see down here. What's going through. This is which note it is so this is the Geo one clusters application node. So this is what the rendered files are. Gonna look like when they get up.

A

This is following this. Is the database node.

A

Main application note- and you can see that we're also a part of this we're going through PG balancer to get the pooling benefit. So by default you know, even if you're not using the clustered form of postcards, you are getting the bullying which we know improves your performance. So that's a already established in here and console these are all trying to go with minimum configuration. We're trying to go with the idea that we don't want to over configure these notes, keep them as simple as possible.

A

Here's database one for the prime primarily has a lot more can say because there's a lot more nodes, that's why it's so much longer application slots is already calculated, and you can see here that that config does gets these. Both the Geo and the pose graphs like we'd, expect that's database to database three, now a little bit about the tech going on underneath here.

A

The way these configurations workers are actually being stored as a as a as a dictionary. So if we look at say.

A

So you know we don't actually template out. Add all these by themselves. We are using this. This is part. This is basically part of what was originally part of the provisioner I've extended a bit to make it basic like a like overlay file system. So app barbie is your generic app when console is enabled when gos enabled these will fold over so here, they're gonna rehearse over. So these eyes, I keep pointing at my screen because I'm silly these items will each get converged.

A

So the first one anything that's in the second one which here is the console RB will overwrite what's an app geo overrides both so basically it's an overlay, so we can keep adding things. So as we extend this, as we add more features, as we add more separation, we can continue this pattern to keep this dry and keep it together.

A

So that is everything I have. As far as you know, what I can show you, while we wait for the play books to run so I'll pause for a moment say: are there any questions.

B

Quick one in terms of the generation of the configurations: how hard is it to add additional configuration bits to this? In case somebody needs to have an intermediate I'm going to add some sort of custom configuration in terms of like the generated, get live, dot RB.

A

Okay, well, when you're aligned, what adding it do you mean dynamically and not altering the run books, or do you mean like if you want to if you there's basically right now, there's no way to inject dynamically, but there say if the run book also has a template. So you could add a single one-liner into the template or you could add the section to the appropriate place like say in the a Barbie.

B

Basically, you could add, you can add whatever you want custom Li to a run book itself, go to the run, run, template I should say correct.

A

Okay, correct at some point actually a great thing to open issue for the ability to have like a user added override that gets merged in after everything else, so that, if someone really wants to override the can, that's probably a really good feature.

C

Right, how would that work with upgrading like if we released a new version of the orchestrator and the customer has a bunch of rides with that carrier? Ava well,.

A

The for this particular piece- yes, because the overrides are going to go into this. Basically, if you, if you list the structure of this like we compare this and.

B

The override is more or what is going to be injected into the gate, like our restructure, less about what like any behavior inside of the orchestrator itself, it's not a change to the code base of Orchestrator. If the idea is to have a method where they can say, merge this over-the-top once you're done generating this right, because.

A

Like if we look at this, get lab rails piece right here, I don't migrate, DB host, that's getting reflected out here! That's how it renders out those key names are actually the entry points, so they're not eros to this page right. So if, if you're following a documentation- and you want to add something- and you know what the top-level key and then the sub keys are, then you could it once I've added the piece you could easily put an override and I can just say this overrides everything else Jason. Would that sufficiently like?

A

Because that way, it's something ephemeral you can just put it outside and added Li before. Is that something that works right.

B

So, like example, cases injecting a CA right, it's not something where you're not going to be necessarily overriding: the PT bouncer, because the BG bouncer is going to be fully installed and configured three orchestrators but say you definitively want to have you know three or four additional c8, because you know your system will eventually be reaching out to internals that are a separate system. The user would supply those parts and then we'd actually have a way to add the additional item to anything and if you love RB, that is needed as an example.

B

It's a great example, but it's a functional example. It's.

A

Actually, a really great example, because one of the next things that I would need to open up an issue for is to is to fit the SSL support in directly, because right now, like the testing is ephemeral. So I can't use the let's encrypt setups, so supporting a fixed SSL and getting CAS n is actually a next step in the next iteration, so that would be a great call-out to make sure that that functionality exists specifically for that as well.

A

Let me add these into the notes here.

A

Jason I feel like we're kind, took a long way around so did I answer your question to the point where you're comfortable or do you have more follow-up? No.

B

That's pretty good for the moment. Yeah.

A

Any other questions I.

D

Like that we're dialing into a specific example that helps if we come up with other specific examples, once we get the issue created, we can even throw them in there.

A

Yeah I feel like thinking through what that, what that ball until there's at least three in there SSL deployment injection of specific things and then specifically getting the ca injection document. So I think that's like three separate concerns at minimum yeah.

B

It's it's a wine case, so it does actually have a number of things to break everything down. It's a good starter issue that is then turned into a bunch of small items. I.

A

Guess I'll riff for a moment since I'm not hearing other questions on things that I've noticed so trying to figure out how to put together a open, an issue and start some Amar's against the actual, the documentation, parts, Fujio reason being.

A

So, like here's jail, documentation.

A

For multiple clusters and.

A

Somehow this escaped it's.

A

There we go so the steps here say to go through and to go through each of these individual steps. But one of the interesting things is that when you come here to like geo for multiple servers- and it says this- but this is not a replacement- this primary role has to be in addition to the other roles that were already set, and this says you know bigger and add the following. Add the following, but this lines a replacement right. So that's not.

A

This is kind of odd, so some of this documentation like when you know going through it, trying to unwind what it means and what's a replacement versus what's in addition to is a little confusing.

A

So that's a kind of a side piece that I need I'm gonna go back through once everything is working, I'm gonna start trying to help with the documentation, so that this is clear for the next person. I, don't know, I know, DJ I know you set up geo before so is anybody else run into that same thing? With this.

E

I've run into okay can I have written to a similar, similar problem where it wasn't clear from the documentation which rules to say yeah I haven't run into this, like I think you probably ran into the primary rule, and you likely also needed what the application role as well on the one.

A

Application role and the Postgres role and like in an Ashes notes, he creates like an FTW and does some sequel injection I haven't found that anywhere listed in the Geo main documentation, so I'm not sure if it's even required or if it's just there, so that's kind of an open issue. I did the issue last night before I was and asked for some other eyes. I bet some geo and some support folks that are following it. So is yeah.

A

There's just there's a couple of things like there are four or five folks that have done various attempts at Geo and there are pieces and some and not in others, and a third of them aren't even in the main documentation. So it's one of those do they need to be documented where they go, what they mean? Why you know? Why are we doing these things? um You know there's a lot of context and that context being lost I think is one of the biggest challenges.

A

It should probably tell why our customers have a challenge with it. Cuz. You know I, it's just there's a lot in here when Eric was in our conversation last week said you know, there's I forget how many hundreds of steps to to deploy this. When you add it all up, even condensed a couple hundred steps or a thousand steps or something that effect.

A

So that is a thing that I want to go through and clean up so that as we go through this until Orchestrator is in GA, the documentation is a little bit cleaner and I'll also help with just you know as we go through and troubleshoot worker straighter and get reviews. So that's just a piece: let's go back and look and see how this is doing.

A

This is still running.

A

Are there any other questions.

B

It definitely have to agree that geo documentation is is foremost and somewhat indirect, trying to figure out exactly how everything works was an absolute pain for me when we did the implementation of the Atlantic G of support within the charts, the resulting Docs, that I wrote it up being significantly smaller than the upstream documentation, and it's very condensed for it. Now. Admittedly, there's many things not happening in the documentation for the charts.

F

Linked to that specific documentation to add up to the depth.

B

But it's amazing how much simpler eleven steps are. Then you know 15 pages of documentation, the come to the actual Omnibus and it's basically came down to me breaking down what those steps actually are. What features actually need to be on or not and what they actually cross communicate. That being said, I don't have Custer involved console in there, and it's I have a very basic book standing up there, Davis Turner's database here so yeah.

A

And that's kind of a lot of the config kind of cleanup came out of your truck documentation Jason. So when I work on this I've got it's on the issue, but I have like five. There are five or six sources of this is how geo should be installed, so I'm, hoping that kind of as a side effect of all of this, that we can align those four with less overall confusion.

B

Right and we should make sure that we are engaging with the Geo team- see what improvements we can have with that yep.

A

A

Okay, so the primary node just got set here. So while this is running, let me go up here and grab the application IPS.

A

So I can start setting myself up to connect to these. Let's see first one: that's the Geo one cluster application node.

A

The second one is prime.

B

I'm guessing in this case, you're setting up puts files I.

A

Didn't catch like the last six words in this case you're setting up like host files, you're doing it off streets. We don't really know what you're doing. Oh, no I'm just copying the IP addresses to a text pad so that when this is up, I can just copy paste and I can go back and look again. Okay, when this is all done, I can run the bench logs again and we can look at effects fetch config and we can look at all those and start working on diagnostic.

A

One of the other issues that we should open out of here that's a thing that I know about I'm gonna, open issue 4. Yet we have a double run of the configuration P of the package piece which we don't need to run because common is injected. The common rule is injected, is injected twice, and that's just by virtue of the fact that, as I ported things over, it was it's there. So that's probably another thing that we should open up as a future here to improve.

A

It is to untangle that so that there's not a double call of a common rule.

A

It's something that wasn't really annoying when there was only a few machines in a cluster when you stand up 15 machines, it's kind of a lot.

A

B

Right and it's a perfect example of if you don't have to run reconfigure on something as large as the other bus, don't because when it's one, it's not bad, but it's 10 it expensive. Yes,.

A

A

Another thing that I would probably appreciate some help, with probably annex like 6 to 4 days when all this works. My goal right now is to get this to where I can just manually add the go1 in the console over to the MAF occasion. There is an API that pointed out in the issue and I'm not very well versed in just curl calling api's I. Don't want to go to all the trouble of writing a Python module principle for this.

A

Yet so, if any one that has you know, experience just calling Perl doing things, Jason I know you do but I know, there's other folks know you're busy, but I could I would really appreciate help in crafting that curl call just so that we can get this minimum but minimum Bible over the line and have it automatically inject the node and then we'll clean that up later, once we have once the you know, we have the proof of concept.

A

Okay, it was a lot of red.

D

A

So that was Gio application one. So that was one of the two modes. I expected. You know just we're gonna bring that over and text we can deal with Gio one in a second.

A

So this is the primary application.

A

You know, let me do this off screen, so I don't pull up every password in the world. I'm recording now.

A

One of the things you'll notice from the last demo is that now now we have a default password. That's here that's set for us that can be that as both I've set the password, but also, if you don't set one, it will give you a password and you can run a playbook that'll show you the password. That's one of the most recent things merged in that was a thanks to the our red team and I had a discussion about some of the stuff with popping up with security pipelines and that's direct results.

A

So thank you to Chris, Moberly and Paul Harrison. They were real great partners. We discussed what was going on when he'd be happening and no thanks to them. We now have an improvement all right, so we're logged in we're gonna head over here, also merged into merged in and not just part of the work in progress. We now have a license, so that's just by default, even with just regular clusters. If you install just one cluster, not geo, you can have your license installed.

A

So that's no longer here that was separate out to a smaller iteration of merge, so looking at geo, you can see that geo is turned on it does understand. This primary note is healthy: okay, yeah. We understand this. This is already there cell peace, current node and its primary labeled main so that all works as we expect. So we've got that much far going so now, let's go turn our turn our faces to the application node. So we are three minutes to time. I'm going to continue going so just you know you need the drop.

A

It is okay because we his a 3-man, so we're actually were kind of Cocoon minutes past, because they're kind of five minutes till but I'm gonna keep going and debugging this so Gio application. One.

G

A

Well, honestly, last night, in my sleep, I dream that maybe we mean I would have put a log for this and there it is all right.

A

A

So that tells me that migrations didn't run.

A

Tried to do a reconfigure.

A

A

So that tells me there's a database high-grade right. That's a database migration for an initial seating right that.

B

B

Can't quite see the entirety of the law, but it's definitely a case where it's literally saying, there's no database I.

A

Wonder if that means that there needs to be a secondary database, migration just.

B

Yeah there has to be a separate secondary database eating behavior.

B

And it does not follow the same thought of magic as the primary database. So when you say hey run DB migrations, it doesn't go. Oh hey! There's no database here! Let me see that for you geo doesn't. Do that see you have to manually say, don't receive the thing.

A

The thing is: we're not using auto seeding we're. Actually, the PlayBook is running back man. Let me make this a that command. Yeah.

B

That does not run the Geo. The.

E

Geodon command is geo, DB I think the initial one is set up and then, after that it expects to be running. Gpu g.od be migrated right.

B

Let me pull up the.

B

Loc- and you can hear the DB migration script in there has the actual do call for when is running into you.

B

Yeah we have a geo DB migrate and it is geo TV set up.

B

We call that and then, as once, that is fully performed, you have to manually call the geogi be refresh for end tables.

A

All if you could paste that in to call doc and then I don't yet I have not seen that anywhere in the docs yeah I knew.

B

That was fun to figure out.

B

It is buried in the docs, but it's in like how to set up the database because you have to set up the full primary and then once you have the actual primary database with FTW turn on and the user created, then you go over to the geo node and do the gob be set up and then perform the geo DB refresh.

B

This is this is buried in the geo database setup and configuration, and that was just painful to figure out.

A

Well, then, I will stand to talk to her shoulders. Thank you.

B

Speaking of this, while I'm staring at it DJ, this structure is QL. Stuff hasn't started, hitting the database, the Geo side has it.

H

Not yet, but it will yeah I know it will I'm. Just like please tell me we haven't forgotten something.

A

But I added heat in my great.

A

Well, that runs I'm gonna, throw out a postulated question that I was thinking about the last last week or so the way that I do secrets right now. Is it downloads the SSH files to look.

A

Set up fails, migrated fails.

B

Right because there's also, you have to actually create the database, it won't.

E

Actually read the.

B

E

Omnibus should have created them.

E

A

Guess that's first, a question like different, so if omnibus is supposed to create it right, what causes it to create? It is the package. Is the package order operations right? The package is installed and then this file is set up.

F

All right reviewing the.

A

Migrations has to be run directly on the database node.

E

So the database creation should have been run on the Geo Postgres. No.

A

Out of this, it.

E

Should have been done as part of the geo PostgreSQL recipe in omnibus.

A

E

What's what's the role on this one.

B

Wrong can set your rule. No okay, geo clothes, grads online. The right, listen address configured with that ok FTW is turned on real TV password is configured.

E

Yeah I would check the logs.

G

E

Reconfigure logs on this one to see what happened it should have created the database.

B

Why is law cursor enabled here.

B

Oh, you ringing all machine yeah.

A

Yeah everything yeah, that's a that was something that uh I asked button geo last night because I'm a documentation, log cursor sounds like it's a rails app, but then it's not a rails app, but I figured out it's some kind of a service. So.

G

A

B

Just wanted to check, because this is not how I said portion alike- that's I'm, just I was like oh that's, different yeah.

A

Setting it to its own node will be an improvement iteration.

B

Yeah it doesn't it's not a hard requirement. You can technically run like five copies of the law. Cursor. You can run on every single database node as that's apology of pair as long as it's talking, wherever the master is yep.

A

So all right, I'm gonna head over to the configure logs Jason if you're go for.

H

A

So here's a reconfigured log side DJ just a keyword like Jail.

B

There should be a task run that actually creates the database.

A

E

Could search for like one that comes after it like? If you look for a refresh foreign, it should have been called, and then we would expect the database before it yeah.

B

Foreign is there at all.

A

B

That turns foreign or we.

E

See above it, though, it that it's uh supposedly created the database.

B

So I see good luck to you, a PostgreSQL user create class database. If you live geo, duction.

B

What user is it trying to use with explodes.

A

User and password is supposed to be there I.

F

E

Notice at the very at the bottom, you actually have the Geo secondary host set to yourself.

G

Here's another thing: I'm seeing right now: bouncer you're, staying up the PG bouncing use effectively the public IP, at least by the hostname, but on the DP instance, you are using the private IP, those a matter of its trying to connect the wrong IP and.

B

Well, in this case, it's literally saying there is no database, it's connected to something, or it would say, can't connect to database so yeah, okay,.

B

I'm wondering if we're like some, how long mean and long the user, using people outside collect Geo so we're connecting to the long database. I mean the geo is PostgreSQL. It should be on different port.

A

Well thing is shouldn't the connection to that to that database exists here in. Thank you, bouncer.

B

There are separate databases there, like, according to the application now technically separate, I've, never used PG bouncer with Geo, so I can't tell you I'm.

A

Just looking at this thinking that know what.

A

That's a great way to find out would be.

A

Let's do something make it Co something.

A

First, get live geo here.

A

So I'm gonna do have that dewy figure. Does that seem sensible to see if we can get that the tank that way.

A

A

G.Od be set up after this Oh.

A

A

Password for your kids, you bouncer no password supplied.

A

Roll kill, fgo does not exist.

B

B

Which database is that connected to? Is it connected to the geo database or is it connected to the host grants database? There's.

H

B

B

So, can you pull up a geo primary node.

A

Geo primary database node right.

B

So the the geo secondaries primary database note that the master database note on the Geo secondary.

B

Okay, I'm here, okay, now, let's look at its left, RV.

B

Those Chris QL is listening or oh, we got the address yeah and.

B

Postgresql insisting on that, so what ports are those processes listening to.

B

Just pop out and run you're in pseudo, so SS tact.

B

B

We've got one listening on port five, four, three two and one on five, four, three one, and since that's listening on the public IP address as opposed to zero zero zero, the geodatabase is listening on four five, four, three one yep are: we sure that the generated configuration is telling it lab to connect to the right place. We want to check our OTT kit lab get lab rails, come on babe.

B

Etc and database underscore to you.

B

That is attempting to connect to localhost 46:43, to which we all not exist correctly, unless you have something redirecting 643 to to the 6131 easy bouncer, yeah, okay, so PD somebody help me here: where do we double-checked the configuration of the running PG master on the node that we're at that will point to the correct port? Oh.

B

Dear so it was I'm not familiar with PG concerts.

A

Like this to get the console, oh.

A

That's broken because of this, so let me take this bit out from reconfigure.

A

Then we can actually take the console.

A

That was error up that way, but.

A

So it's not going to change it directly. Let me think you got your eyes.

A

The file is pinkie, bouncer, I and I, and I always forget where it puts it.

A

There's the eye on us.

A

A

What goes to the bouncer datadirect leadership.

A

A

A

At the right place for this feel like it's somewhere else,.

A

A

I'm just doing what I would expect.

A

As the admin users for both.

A

Does that migration have to get run over on the database? Oh it's kind of here. It.

B

Should be run wherever you can actually reach the database.

B

Technically, you can run it from someone else. I I have always traditionally run a little safe post again, I had been ranked cluster of a kind. I have been using PD balancer.

B

Yeah you take the line that Ross, unfortunately, Ben doesn't understand.

A

Databases, the 9i.

B

That's probably generated yet somewhere in their.

B

Own by root, readable, like it lat.

B

Just pull the options for databases I&I or the attributes for eg bouncer databases online, if not nil, so options is part of the call into the initialize and then something breaks their database. I&Amp;I is new or the attributes from the omnibus laptop RV are coming in as brokenly nil.

A

A

This is the acute so.

F

B

Where, where is the database ini attribute? It's just databases, so we have to look at what that is actually generating to, because in that issue.

B

Right now, I'm, looking through the documentation to try to figure out how this is supposed to be configured so.

B

Supposed to be reading yeah how you begin to point we're supposed to be reading the databases, dot JSON to try and pull that endings. It's exploded before that.

A

I'm, just gonna try something really quick: I bounce bounce err.

A

B

Don't think that the bouncers problem, that omnibus code is exploding.

B

If you look, T sequel doesn't have a methodology to reach out to Q.

B

B

There's that file exists.

A

F

That I and I know the octet level check Harrogate lab people that are see.

F

On the application, though, is pop we're not okay, see I get let PCWorld probably not tell function. In that case,.

B

All right so can you show me then get led to Harvey for this application. Note again.

B

B

So there should be nothing about your geeky bouncer databases, entry, but what you had in place that should have had any problems whatsoever, that's causing the explosion. Something else is causing the inability to love the database, configuration options.

B

So what you had added here to get to the get lab HQ geo production. It was not the problem.