GitLab Quality Department, 23 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Troubleshooting Quad Planning Dashboard

Description

Tanya Pazitny and Kyle Wiebers troubleshooting Quad Planning Dashboard

A

B

A

How's it going okay.

B

A

No problem, this is I, don't know it sigh sense and the way data is imported, as always like. There's always something new, so I know that it's super confusing and I'm not really clear. What's on as well yeah.

B

It feels like the separation between the underlying data and, what's shown here, like you, have a reshef refresh button, but it often is just it makes you feel better by pressing it, but it doesn't actually do anything right.

B

A

So so I I think I know what what is happening. I want to confirm kind of like what what I understand here so like what you're saying is like this issue is here. But when you look at the issue, it is not ready. It is like complete or complete, though action.

B

An example here since had mentioned to me that he finished them all, but I don't know if there are other ones that this is also applying to yeah.

A

Probably so, let's let's talk about like how the data ends up and like what we're looking at here. So it's like there's a production copy database that data is imported into the data that is read by sizes, so you're, like three steps removed from what you're looking at on, get lab, calm and there's the potential for error in every single one of those steps.

A

So what we found is sometimes data doesn't copy correctly from your lab comm database to the production copy, which those are both managed by infrastructure and then sometimes there's failures importing from the get lab comm copy to snow. What you may hear called snowflake, which is the database that is read, advice, I sense.

A

So usually when you see something that's off from Capcom, even when you, like click refresh the the problem, is somewhere in that chain and then I guess. The other thing to add which we had already talked about is data is imported I'm, just going to say every 24 hours data is imported into snowflake. Every 24 hours from the production copy says every six in the handbook, but it doesn't seem to be, and then this dashboard that we're looking at is also refreshed every 24 hours automatically.

A

So the data would need to come in through all of those different chains and then and then the dashboard would get refreshed. I think what's happening here. I'll just bring this in is so like here in the data Channel.

A

Said that there was an issue with airflow this morning affecting all data extractions, so that's that last step of copying from the production copy to into snowflake, so I think as I looked at it. The thing that jumped out was I.

A

Just don't think that data is there and usually what happens is the data team will do a catch-up and talk about it in the data Channel, but I haven't looked close enough to know for sure, so we're gonna take one of these memory issues and just write a query to see what are all the labels that are currently on it. I think I scrolled, that's it but we'll take like this one I'm just gonna copy it have. You tried querying size sense at all before no.

B

I haven't this is okay.

A

um Well, if you log in, do you have all of these, like all of these buttons over here, specifically like new exploration? Yes,.

B

A

So just gonna talk about like three tables that are probably the most important here. So the majority of the data that you'll be looking for is like get lab comm. If she'd like in this case we're gonna be looking at issues so Caleb comm issues XF, which is in the analytics, you could say, schema if you're familiar yeah. Okay,.

A

All gitlab comdata is imported, so it's not just get lab and get lab comm. It is every issue created by any user of yet lab. Comm is in the stable, um it's sanitized, so it's like scrubbed, anything, that's private is scrubbed. Anything that is publicly accessible is what's important, I should say so what that means. We have to like, be a little tricky and how we, how we do this so there's an issue ID an issue I ID. So what I was going to do is just use issue ID here, which is the like.

A

The reference for the.

A

Or the like in the merger quest copied the merge request URL so like this would be the issue I ID, and then we also would need to say the project ID, which I can't recall, get labs off. The top of my head.

A

You will I was playing around dark mode earlier, definitely liked it. I mean it's great. It's a nice first iteration I should say: there's some some things that are just a little too too much contrast, okay, so here's all the data.

B

A

Don't really need to select all of it. What I was going to select is labels we'll just do like.

A

She tied off and we'll just do labels for now. So just like these two fields, you know what we should also do like last.

A

All these feels issue updated that so there's different conventions. So sorry, I, don't know if you saw that I was just looking over here to see what it is but issue up, there's different conventions. Sometimes it's last updated at across different objects. This time it's the issue update to that the last time it was updated was yesterday at 3:00. So if we look at that issue, don't know if it's anymore GMT.

B

Is that probably that lines up with I think when grant.

A

uh Yeah, maybe I think it might be local time, we'll see so I think sighs cents stores the stores everything in GMT. Why is this? Like? So sighs, ten stores everything in GMT, but it shows it to be.

A

But I wanted to do was open the issue and I guess I'll. Just do it this way.

A

And see and see what lines up like what activity lines up.

A

So the other thing we'll look at really quickly is you have to hover over to see all the labels here, so you can see the eat? It's really small, but it's not listed here. That's what's powering the charts, so it's really the chart is just saying give me all the issues that have quad planning ready it does. But when you go to the actual issue it doesn't so if we go 618 at yeah.

A

Yeah I mean there was no other update yesterday, so this is what's weird yeah.

A

Right, there's no reason why it would be last updated yesterday and not there. So, let's I'm gonna do one more check. There's another label stable.

A

Things called the label Thanks.

B

So why are you doing this? That's interesting on my side. You have data and that's where you have analytic staging analytics. I have views, I'll, get lab comma shoes when.

A

B

To search I've been trying to type the same thing that you have I, don't I'm not getting any results. I'm getting errors do.

A

You have do you have so like if you do. You have like this button right here. The plus button for dashboard.

A

B

A

B

I, don't have the data that you have there.

A

What's your is your, do you have like this this one selected right here or it says a get lab.

B

Thank you, yeah.

A

A

Because what I'm gonna, what seems like, is labels you're, right, you're, running into a frequent pain point of mine, so we rely on labels. A lot for information and the import process for labels is super unreliable because again it's every label every project, every issue across gate lab comm. It's just a large volume of data to process, and it it doesn't work.

A

So I've been trying to work with the data team to say: can we just get a more reliable import over the data that we use more frequently and it's not as not as urgent of a priority from everyone else, which makes us like target ID equals.

A

Yes, I'm just going to start with this cookie, you.

B

A

A

So I'm highlighting the query because there's like a snowflake IDE that I don't have access to where it's like they get different place to write, queries against this data or you can do it like at normal, like you would a normal query, but I don't, unfortunately, have access to that. So it's kind.

B

A

A pain to write like have two queries at the same time that you want to keep.

A

A

So what I'm doing here, the I'm just joining the labels table back so I can get the masked label title.

A

And then we can have a better sense.

A

Of what's going on yeah, so I think I'm going to ask in that issue.

A

Is it looks like.

A

This is just one example. The day team does like very specific examples, so this is great that we're we're finding this, but it looks like the label last updated or the issue. Last updated was changed from the comments.

A

However, the label wasn't applied, so I'm gonna ask in the issues that they created.

A

A

Just want to review if this is clear.

A

Yeah, okay, so they're still back going the lab, calm, dbx, very.

B

So they'll just do it manually to resolve this yeah.

A

Yeah so I voted the issue just to follow along and was hoping that someone would say, but for now I think the guidance is gonna, be let's wait and see how this outage there's at least one of these a week. I'll say so. That's where it's not an infrequent occurrence is.

B

There a way is there somewhere, where there's a table or a view where it says you know. Snowflake was last updated blank.

B

We can see on the dashboard when it was last when there refresh button was last push, but that doesn't tell you anything about the underlying data. That's.

A

A great question.

A

Not that I know, but let's try and find one.

A

Let's ask in the data channel.

A

A

A

A

A

So I'll say: is there a table that can tell us when the last successful gitlab Convy be in court occurred? If there is a suspected De, Palma I'll say to help troubleshoot the suspected data problem, I.

A

Just see CG on that, there's not one that I'm aware of I'm, probably not going to go searching for it, just because I think that's inefficient, I'm sure there it's like I bet DBT audit log like we could probably pull something out of here, pull something out of one of the other two tables we'll see what they say, though,.

B

This even just understanding now this exploration is really useful. Thank you. Yeah.

A

Yeah no problem I think we need to do something like this with every like all the key items, because the more and more that we're gonna be using science, the more questions are gonna come out and the more that you kind of have an understanding of the data model or at.

B

Least, are able.

A

To explore the easier it's gonna, I'm, not gonna, be like a bottleneck on everybody will be help. Everyone yeah.

B

Definitely and I'm excited to hear what they say about when you can tell when it was updated, because then you could self-service and say: ok, it's just. There was a problem. The last time I was updated with 7 hours ago. I'll just wait. It would be great if there's a button. You could push to do it yourself, but hopefully they tell us how we can know.

B

A

Yeah I usually look in the data Channel and then I'll usually ask a question and then, if someone's like yeah, it looks like there's a failure, then I'll usually create an issue. So like that's my normal workflow, it is terribly inefficient, so I'm looking forward to what they say as long.

B

A

Problem, sorry, it's sorry that we can't actually get the data correct, but at least now it's clear like there's. There's a lot of steps in between they're clear on the steps in between you'll have a comment: yeah.

B

Thank you. Okay,.

A

See you Tanya have a good one.