GitLab Application Performance, 15 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab.com performance analyzed via snowplow

Description

The purpose of these dashboards is to show the performance of GitLab.com for users who allow us to collect performance data via snowplow. This can be used to determine where there are potential performance issues in the product in order to improve the overall user experience.

Handbook page describing dashboards: https://about.gitlab.com/handbook/engineering/development/growth/product-intelligence/gitlab_com_performance_dashboard.html

Issue for feedback: https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10839

A

Hi, I'm wayne haber director of engineering, and I want to talk about some new gitlab performance dashboards based on snowplow.

A

So today, one of the primary ways we look at performance of git lab is through uh long largest, contextual paint, which shows the amount of time that it takes for the user to actually see the page if it's painted it may not be the full page or all content, but the primary portions of the page, and we compute this via pulling it lab for specific pages for a specific user. This is a great way to look at performance.

A

However, it's missing some things and by design where it's not looking at performance across different users or at different times, um and it doesn't give us a lot of measurements.

A

So we can't do things like a compute, a 99 percentile, because we don't have enough data points to do that, so um what I've created with help from lily and christopher and others is these dashboards based on snowplow, where we have a portion of our usage um by users in snowplow not only for usage but also to collect performance information, and this is for a large number of users and we get a larger data point. So we can compute percentiles.

A

So I've got the handbook page here and linked off of it. The first two dashboards. The first dashboard is the overall performance dashboard and it shows the average time a number of different things over the last three weeks per week, the average time um and also uh various percentiles, including 99th. So generally, the slowest of these will be uh or long largest, contextual pain is less than the total um load time uh as computed by snowplow.

A

So the average time is actually pretty good here, uh even 90th percentile less than 2.5 seconds it's 99th percentile, which um is a bit high. So it means one and a hundred are greater than seven point. Seven, seven point seven seconds ish. So I looked at this. We also looked at this in terms of page hits. Even when page hits are lower, the 99th percentile is about the same. So it tells me that performance is not related to the amount of pages we're getting at that time period.

A

It's an interesting thing to look at. Nonetheless, then I looked at the average for all metrics, where the um total load time according to snowplow comes into play, and I I I deselected, the ones that are really low to make this here so average gives 700 milliseconds for redirect 700 for processing 500 for dom interactive to complete. It looks like we had a blip here in app cache.

A

We went from really low to uh relatively low at 100 and change milliseconds, but we're not really as concerned about the average, because you don't see these very high, it's about the 99th percentile, so the 99th percentile 4.3 seconds for processing um 3.3 for interactive to complete 2.7 and change for redirect and uh 2.5 for request. So what do these things mean in terms of snowplow? So let's look at processing and dom interactive to complete.

A

So I have these pages here, uh processing time since processing time dom interactive to complete is time for current document readiness to change from interactive to complete. um You can see the definitions of these here and also a picture here. These pages linked off of the um handbook page some potential things to look into overall, the yeah there's a redirect on a bit higher than I would expect on average um the 99th percentile. um So you can also see over time per hour. So you can see it's a bit spiky at times.

A

So most of the time that you know, 99th percentile is in the seven and a half change uh seconds, but you know occasionally it does fake up. Here is that 13 14? You know 11 up here, etc.

A

um If you're looking in terms of page hits, sometimes when there's higher page hits there's the performance is not as good, but that doesn't seem to be very correlated similarly to the others. When I look at page hits um not correlated to performance, so you can look at pay per day of the week, so we actually see slightly worse performance on on 99th percentile on the weekends.

A

On sunday and saturday versus during the week- and then interestingly uh I found is we see when, when we have higher usage, we don't see worse performance, it's actually at hours, zero, utc hours, zero and one, and then all the way over at 20, 21, 22 and 23. We see uh 99th percentile a bit higher than other times, so maybe we're running uh maintenance, jobs or other things at those times because there's lower usage, but it actually gives user also lower performance during those times.

A

So it's something we may want to look into something great to look at it across all pages in all routes. What about specific routes? What routes are the ones we perhaps should look into on a route by basis? So here we have a table of some of the top routes, such as view file, merge list uh source code, mr issue etc. So you can see the percentage of page hits for the percentage we're collecting with snowplow for the users that allow us to do that percentage of unique domains, so not just hits, but how many?

A

How many people are using them and then average load time percentile and then up to the 99th percent? So let's look at, for example, merge list. Seven point: seven percent of page hits pretty large portion around sixteen percent of unique domains. Users average load time according to snowplow 1.5 seconds. You know, percentile 50th percentile is pretty good, 90th percentile, 99th percentile 5.7 seconds and interesting um and for a highly used page.

A

So, let's, let's take a look at that, and this is the next dashboard which is linked from here about drill down dashboard and also linked off of this one at the top.

A

So what I did is I chose went over here. I then chose which one do I want to look at, get that merge list.

A

And, let's take a look so average time per week for the last three months average, you know uh it's pretty good. You know less than 2.5 seconds. Definitely 99th percentile, as we mentioned, you know five and a half seconds. um What are the? What are the things that that that that total time is comprised of top things requestable um at an average at least 800 milliseconds processing time 500 redirect time 500 dom interactive to complete 400.. But again we don't care as much about the average. Let's don't you look at?

A

Let's look at 99 percentile, well, 99, percentile, processing, 2.6 seconds 99th percentile, request 2.4 redirect 1.7 um dom interactive complete two seconds, so some things to potentially look at if we want to make this page uh faster for the 99 percentiles, look at processing, perhaps processing time and request time again. Processing time is crossing time request time time between the user agent, sending a request and receiving the first button response.

A

So a lot of server time of computing data before it's sent down to the browser is a part of that um look at per hour over the last two weeks for this paid. For this. um This route, um you see it's kind of spiky as well uh by by that hour over the last two weeks by day of the week again, that's sunday through saturday, nothing different there than we didn't see before, and then I have for gitlab dash org project. um What are the slowest pages we've seen over the last month?

A

So you can cut, and these are all clickable. um Then you kind of see well and it's sorted by the the total time that it took on this one. So this one took, for example, uh this one took 20.5 seconds. Where was the time spent in that?

A

If I go through that well, these are all really small in milliseconds request time in milliseconds, 20 millisecond, maybe 400 in processing time milliseconds is the top two. So that website is. Why is the request time? Why did it take 20 uh 20 seconds? If I click on this.

A

I'm going to take, I think I loaded this one in advance. It does take a bit of time to do this search. um It's still loading there. um This is what actually returns. So here we go, so it didn't take 20 seconds. This is actually the one I opened up a couple, but it did take a good bit of time and it is resulting in a good bit of data. 21 pages of data computed.

A

um You look at a different one here and let's look at one that took a lot of time. We said in redirect time, so if I sort by redirect time this one, the slowest one in redirect time, at least for this- is 300 milliseconds, not a lot of time. So if I open up this, one here came up a lot faster, so I'm not sure the reasons why? But this gives us good breadcrumbs to look into to look into performance.

A

Look at things that may we may want to spend time improving, but there's a faster that things are the happier our users are thanks for listening and have a great.

A