GitLab Data Team, 19 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Taylor Murphy and Justin Stark Pairing

Description

Justin wanted to get a sense of how I work and use GitLab. So I recorded a 1 hour session and tackled the following issues.

Provision Derek https://gitlab.com/gitlab-com/access-requests/issues/2050#note_218331402
Create analyst_infrastructure role https://gitlab.com/gitlab-data/analytics/issues/2482
Upgrade to 0.14.2 https://gitlab.com/gitlab-data/analytics/issues/2461
Re-enable source freshness https://gitlab.com/gitlab-data/analytics/issues/2435
Organize Macros https://gitlab.com/gitlab-data/analytics/issues/2455

A

All right cool, so there's a little Google Doc. If you wanted to you.

B

Looked at it yesterday, yeah.

A

Awesome so what I figured I was gonna do is just kind of go through some specific tasks that require random steps, and you can kind of see how I work and then just ask questions or we'll go from. There sounds.

B

Good I'll be quiet until I have a question and then, if you there's anything, you want to call out feel free to call it out, but no pressure either way cool. All.

A

Right so this first one access requests. Bane of my existence, all right, so I, don't necessarily need to follow this directly. Cuz I. Have it mostly.

C

A

So there's a section here on how to do it properly.

C

A

So I need one of the other things I need, because this is going to be touching a config file. I need an appropriate issue in you live as well.

C

A

All right, so, when your security I'm in you can see the account stuff can't see billing just.

C

A

A

You use looks like old, password generator. I will put this on the screen. Obviously, but I mentioned I make a temporary password and then send it to them, and then they have to change their password.

A

Login name has to be their email or fill out an email because we didn't log in via octa. So it has to match that I.

C

C

And then no defaults.

A

So he's it's his new user account. They also get a.

A

Role, the reason we do individual user roles as well, it's just for basically provisioning, there's something that doesn't fit into a role that we need to grant specific access to.

A

We can grant it today that specific user role and that would be documented as.

C

A

So this has to be granted to.

A

Him and then he's going to be analyst core.

C

A

Cool, so that's basically done from the snowflake side so from here I like using the web IDE for small things like this, because I don't do anything locally.

C

C

C

C

C

C

C

C

Yet would akela calm, confirm assignments, go.

A

All right so I've got. Is these roll got his.

C

User, any member of both of these, though really I, should switch the.

C

Okay, so he's in stiff-leg.

A

Looks good looks good so happy with that March that and then I use one time secret to send your credentials because I don't want to put them in one password.

A

Well, I'll take it out the screen.

A

And then I just DM.

C

Their credentials.

A

Boom, alright, so that's done so I will cross link.

A

A

Sign myself subscribe.

C

Comment edit, this I know I'm just click. Yes, yes,.

C

A

Okay, cool, so that's all done. I I reload, usually just to make sure everything stuck. You clearly didn't.

C

I've already done.

A

How do I'm getting that air.

C

Just do live snowflake, yes,.

A

Yeah! Okay, cool, so that one's done.

A

All right, I'm actually going to skip the infrastructure. Well, now, I'll. Do it I, don't know why it's.

C

So this yeah so I'm done with this. This.

A

All right so new analyst infrastructure role.

A

This one will be fun.

C

So I think I get to spin up Deltona to try and do this all.

A

Right so we're gonna make I'm just going to make it a copy of analyst core for now, because basically, everything.

C

Let's just go with that for now.

A

And then actually I probably won't fully provision it and I can show you what I would do just so. We can get approvals and everything on this.

C

Infrastructure.

A

Okay, so now I'm actually going to.

C

Let's do this over here.

A

So the permission bot runs.

A

The end like a melt, oh no project and we'll read through whatever, however, you've got configured.

A

So what I would I've been doing is basically I have a small version of the schema because it could take a little while to run so I have a small version of the schema and I would just replace, basically whatever role here that I'm actually interested in. So in this case, so I haven't actually provisioned.

A

Let's actually go ahead and do that so I can go ahead and make the I can make this role and just not give it any grants. It won't.

C

C

Infrastructure.

C

Cool, that's done all right, so.

A

So I have this config small Hamel definition.

C

C

Need to do, there's like certain weird quirks with this I want to get it all automated.

C

There's that their users I think if I just here into this.

C

See what happens? Analysts analysts, all those that should.

A

Work so now, if I come to the command-line, go through my history, I.

B

Do that same thing, all the time and I also feel a little bit ashamed when I do it just like.

B

A

All right so I'm just running the command I'm, just gonna help put it to God I have to.

C

There we go, I need my virtual environment and now I can.

A

Okay, so now it's running, you know what it's I'm going to run it in that directory yep.

C

You get rid of some of these, so I can actually see what the hell enjoyed it's funny to be in the lab in there.

A

A

So the current permission bought, it only runs in dry mode. It won't actually execute them. It'll give you a diff of the permissions that role currently like the things currently have versus what they don't and find. So we can see so it's loading the spec file, checking it for errors. It does a check that it all exists and then it will actually like query. The information schema to get everything and I don't know how to make it.

C

Reload, while I'm viewing the thing in here.

C

A

Still going, but we could see in snowflake.

A

So it uses this permission, bot user, which maybe I shouldn't be doing locally. But yes, it does like show roll show users and then it just kind of parses through all that.

A

Thanks so fetching great privileges, nice here we go so these are all the grants that should be run based on the definitions that we've had. One of the updates I did was to do all tables and then future tables, and that's what, when it's defined like, so what we do dot star on either the schema or the table. It just doesn't the krandall kind of like you, expect so I'm going to leave this M R as it is, and.

A

Get approval I think we probably should get approval for new new rules, new roles, okay,.

C

So I'm going to.

C

C

Look all that one done for now: artists.

A

All right, how are we doing that time, great less than 20 minutes, so, let's move on to this upgrade so.

A

14.2 is a very minor release that has a like two minor fixes: one. This dbt snapshot was hanging and then this one I don't even care about, but I have an open issue for reinstating that job. Oh yeah, this is this is actually related, so we can actually do this simultaneously.

A

So this job is.

A

Shut off right now, because if I look in the actual project at the dag, oh.

C

A

Already looking at it, so the the way the current dag works. Let me actually do this.

A

Better visual for them.

A

A

So currently, every time DVD, the beauty runs, it should broken pipes. This is why we need to you know it should run source freshness and then it'll do a db2 run and then they'll do DBT test. So the source freshness is nice because it can like say hey. This is out-of-date or warn you that it's getting out of date, it's not great when there's some models that just genuinely aren't updated- and you know like a 24 hour, out-of-date thing and it's like well, no new data actually exists, so there's some false positives there.

A

But what I've currently done with that job is source freshness I'm just doing true, because it was hanging and it was taking all the airflow resources. So we can simultaneously test the 14.2 upgrade and.

A

Re-Enabled this job, but the oh actually I'm, going to instead of making a new issue in the infra remove this to the infra data infra project.

C

And then I'll open up NMR on this one.

B

Do we have separate so many separate projects just for permissioning or what's.

A

I've talked to Thomas about this I'm, not a huge fan that we've moved the infra wrote out. It was mainly for having like different CI jobs for actually building the docker containers. I don't buy that it needs to necessarily be separated. There's Thomas has the same permissions on this project as he does on the main analytics. One I would like to see this project come back in to the main main main repo.

B

Make sense that sort? That's what I was reading yeah.

A

So literally this this, we have like three images: there's the airflow there's a data, then there's just a DBT image. Thomas has more context and why he thought that was a better idea. I think is just precise, but so I'm gonna upgrade this one and the nice thing about this is.

A

So it's going to part of the jobs it will automatically make the docker containers and then what I can do in this? Mr, is test those and mr which.

C

So I'll just use I do I, didn't update the deck.

C

All right, so we get rid of this and I will test it.

A

Should be pulling the latest one, so I know this command works. We have. Oh I need to make a separate issue to change that. So there's this there's a separate target in DB T, because this variable and the profile you're not able to pass as a command line variable. So there's this client session keep alive. So there's long running, queries for like snowplow transformations, I thought this was the problem, but why that job kept hanging? And so I made this separate Argus it kind of was the problem.

A

No, it wasn't a problem, so I think I. Think I can get rid of this, but I'll do that in a separate issue. So anyways keep this job here and then I can actually look in the CI CI that yeah Mel. So these are all those jobs that are available fur run in a merger quest, one of the jobs that we have is the source freshness, and so it runs DBT source snapshot, freshness, and so what I can do up here is I'll. Just update this image to pull from.

A

Whichever one it's actually building, I can go. Look at the containers.

A

And it's not done yet, but it should just be the branch name.

A

This this is a I'll revert, this that were actual revert, but I'll revert. This change I'm just for the purposes of testing them.

C

A

All right so now that those commits are available in the merge request. I'll go back to the mr and trigger the analytics clone job so that the database is available for these jobs to run against.

A

These ones that are running automatically when you make any change to a Python file, it'll run my PI I tests black for lending, but that these aren't available so you're not changing anything to any Python files.

A

So this takes a little bit, but we gotta wait anyways, because this job needs to finish.

B

I'm, like really stoked, to take advantage of these good lab features, but also totally overwhelmed by it. It's.

A

A lot to get used to I'm there's some really cool things and I forget how much like stuff there's the build-up is, there's like all the hotkeys. You can hit it all the keyboard shortcuts for doing things with your keyboard. This is a shift question mark this up here. This is really cool. I should actually show this too I think it's in our onboarding, but this is like the performance bar, so you can actually see these are all of the queries that are run to load the data on this page, which is super helpful.

A

If you're trying to understand the gitlab, a common database and what's actually happening, and then you can see like you just see everything that's happening so I have this running all the time: okay, cool! So the build is done or refresh this. We should have.

A

Yep, so that's what it's called and.

A

This is still going.

A

Now Lord at three remain simultaneously.

A

We'll see how that goes,.

A

Alternatively, I think what.

C

I want to do this locally yeah. Let's do this locally. Okay,.

A

I can also we'll just let that run so.

C

I'm going to check this out, I don't need that.

C

What this light bottom half.

C

All right so now.

C

A

Yes, cool so we're good and happy guy, so I'm going to first I need to export the git branch.

A

Interesting I guess these are the new updates. I might need dad I.

A

Wonder with my DBT path profile that variable set.

B

Yeah I'm trying to get out these sorts of environment, stuff setup and I have not figured it out yet, but I haven't had time to really spend on it. Since I learned the last things that I learned I.

A

Think I should be okay here, because this actually just I'm not actually running DBT on this okay, so airflow isn't running I'm gonna trigger a job locally and.

A

Don't run in kubernetes I hope, but it's going to run. It wants to run against that warehouse. Anyways.

A

Okay, so this is like my local copy of.

C

I think it should trigger.

A

This is something that, like it'd, be nice. If the analysts could do, but there's some bugs and stuff with it, yeah, because I soul sometimes have to like pop into the gcpd admin area and kill a node or something but yeah I don't have to fully wait for this to finish running cuz I'm worried it won't with three simultaneous cloned jobs running on the same box.

A

Yeah they're taking longer and longer I.

A

Think Thomas has that open issue to think.

A

About smarter ways to clone.

A

Curious so to 11 minutes raw.

A

Its analytics analytics, so we may be here for a while.

B

A

Is one of those changes where my patience gets? The better of me I've been I've been running 14.2 locally for a week. It's been totally fine.

C

Let's see what's actually running here.

C

You doing airflow.

A

Let's see it decided to spin up to.

A

I'm, like a about airflow, it's it works. Well, it's fine! It just could be such a better user experience. Oh, it actually is working boom. Here we go. Oh no! It's not working it's erroring out because it's not going to be able to find the database.

C

A

That's that's fine, because I'm not I, don't care that it's like I, don't care about the data for this instance I! It's I, just wanna make sure it's! Oh and now that I'm thinking of it, we actually cloned the raw database, because it's actually looking for the branch name raw and that won't exist. But this tells me enough that the job works, airflow is happy, and so we can. We can call that done.

C

So I'm actually I love. It.

A

Alright, so resolve working progress, so we're happy with this.

A

So I'm gonna merge that change. It should automatically build the latest version of each container. Yep. That's what it's saying here: I.

C

A

Revert back that one change.

A

So that's done before I forget I need to make an issue to remove.

A

B

I'm unmuted, so you can hear all my shuffling yeah. Sorry no worries.

A

Cool so I'm happy with this, so I'll just set this to merge on pipeline succeeds, you ever see somebody say set to M WPS, that's what that means merge when pipeline succeeds cool. So that's done. So that's going to close that issue.

A

We've done this and.

C

You know pipeline succeeds.

A

And I don't know if I actually want to do this one yet cuz, it's gonna.

C

Make some conflicts.

C

A

Come back to that yeah I made this one real, quick cuz. Our macros right now are.

A

Kind of just all over the place and we've started to have a little bit more discipline about a great point and like in the tests, folder organized by source model type snapshots, we're doing the same thing so I like all these macros are getting a little out of hand.

A

It should be fine yeah, because we've got UDS there. Okay, this businessman talked myself into it.

C

All right, what's here, I.

A

Like BS code I've switched recently from pycharm, this feels like a little bit lighter and a little bit more fun to use yeah.

B

I'm using the exploit too so I'm also, you assume that since you're talking to yourself well, I guess it's talking to me, but when assumed that like that's not because of the pairing session, the best just how you work, because my only observation, so it's.

A

Actually, don't only when I'm really trying to rubber duck something, do I talk to myself, but I'm, literally just vocalizing, what's going on in my head, so this is like a stream of consciousness. I think no.

B

I appreciate it I just thought: I'd.

C

A

Of these are old, I'm, not even sure this was actually in use anymore, but it's another another problem.

C

So I want one called.

A

A

C

Make a utils one.

C

C

Trying type this is for surah.

C

Case when doing int detail to me, please.

B

Don't ask me again.

C

C

Let's make a product.

C

Yeah praise, it's actually, that's fine! We'll do product I.

A

Think so these are up for debate stage mapping. This is for product as well.

C

C

mmm I should name it pings.

C

And then this one should be.

A

So this is for pings.

C

C

Support SLA.

A

Is Zendesk makes me feel like I should be a twitch streamer now, except there might be some security risk streaming like warehouse all day.

B

Honor, if anyone doesn't like anyone, does anything like that, there's.

A

Definitely people that stream I don't know about a good lab, but I just know in general, like.

B

Stream, their development process yeah.

A

There's some some fairly popular twitch, streamers I, think I.

A

B

No idea I've never actually gotten on Twitter. So that's a reason.

A

Well, it won't be good name for this warehouse. These are like warehouse common things, so granting altering.

C

We can generate this.

A

Like I'm, unsure I just make a decision, and my failsafe is so we can iterate I've.

B

Already started already caught myself doing that and I'm like actually pretty actually kinda like it in.

A

Its a bias towards action and results.

C

Is project included in engineering metrics? This is very specific.

A

It's something very specific to different business units. They don't necessarily map to the data models.

C

That's fine action type, so this is Gil mmm calm.

A

C

Internal paired namespace, that's good luck and then finally, product category I really think this is aura, but I'm not sure it is.

C

C

Yes is used, hora.

A

A

Sometimes use this is sublime urge, which is a nice little tool for like seeing what's visually seen what your local trainers are you to be the history for different branches, and things like that.

A

All right, so we're set up I'm going to instead of waiting for branches around just gonna, run something locally if we can find the there's a bunch of so. If you look at the Deepti project, animal there's a ton of macros that get called in here. So if this works, then everything should work run.

C

See if this actually.

C

This actually merge so that it's done.

A

Oh helps if I staged all the files likes.

A

Yikes I just staged all the balloons. That's.

B

A

B

Feel like directory changes are like the most I have the hardest time, keeping a track of. Can you try that stuff in my head about each other kinds of changes, yeah.

A

And it I feel like we're not doing a great job with showing those discs, because it'll show like a file being deleted and sometimes it doesn't properly match name. This seems to worked fine, because the names are all all good Oh.

C

Interesting, this readme got moved accidentally.

C

This readme is for the UDF's.

A

Just want to stage working directory and entry.

A

This so this was very happy. It ran all the macros, so I am fine.

A

Somebody was sending this, mr to me, I would hold a little bit higher standard, but I will document it briefly. What changed here.

C

Hopefully, logically,.

C

Looks good we'll check once more I'm sure they've missing.

A

Yeah looks good I think, ideally, I'd have maybe should have.

C

On the phone, what is happening.

C

Where's my idiots see this am I going crazy.

A

It's not showing the disk properly. That's really weird, but locally I have regex, and this one.

A

So you need something here all right: I.

A

Think we struggled with directory changes. This is really interesting.

A

It's like it doesn't see.

A

But it's in get history.

B

I've got no good advice, also I think I'm gonna jump off and go to and when word of this key review cool but yeah, this has been great. I got some good notes and it's been really helpful, at least for me. I hope, I hope I can pay it back. Somehow so yeah.

A

I'm gonna fit on this whole problem that I'll upload it to YouTube for all the world to see sounds.

B

Awesome, alright thanks man, alright.

C

This is really weird.

A

And it's just the readme is not being there.

C

All right, well, I'm, gonna.

A

C

This out there, thanks for watching.