GitLab 14.6 Release Kickoff, 17 Nov 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Enablement:Global Search - GitLab 14.6 Kickoff

Description

https://gitlab.com/gitlab-org/gitlab/-/issues/337995
https://gitlab.com/gitlab-org/gitlab/-/issues/284139
https://gitlab.com/groups/gitlab-org/-/epics/6974
https://gitlab.com/gitlab-org/gitlab/-/issues/300081

A

Hi, I'm john mcguire, and I have cz with me and we're with the global search team and I'm going to take you through some of the updates that we have coming for get lab, 14 6..

A

So the first thing we are really focused on two different objectives. As we look at 14.6 in global search, we want to make some incremental improvements to the ux and front end as well as really focus in on performance improvements, specifically looking at how we can increase the performance and reliability of of gitlab.com as a sas.

A

So starting with the ux and the front end components. We have two changes. The first is that we want to make the the options that are on the left-hand side when you do a search that allow you to filter on status and confidentiality.

A

Today these are radio buttons. We want to switch these into multi-select and make them check boxes, and so that would be a change. That's for issues and merge requests today, but we would carry this forward as we continue to expand out scopes in the future that have these filtering options.

A

The next thing is something that we kind of saw can be a little bit confusing for uh customers. Sometimes when you are searching- and you get a option to choose that search term in the autocomplete as it's focused in on a group or across all gitlab, and the same thing happens when you're looking at project. You can then also get an offer to search that term specific to the group.

A

Those contexts are really helpful, but they can be a little bit confusing because the text runs together, so we are looking at breaking it out to design. It looks like this, where you can have the group and the list of the project out to the right-hand side, and then it would also be reflected in the search bar as well, uh if you're searching in that context already.

A

So this actually condenses the view a little bit, but it also creates a better separation across the alignment where these group and projects are listed so that it's not confusing or blends in with the search term. That's chosen, so the expectation is, it will encourage more users to actually drive into their scope or group from the initial search, which should lead to a better experience.

A

And so cz is going to take us through some of the performance related objectives that we're looking at in 14-6.

B

Thank you, john one of uh performance improvements we are going to do in 14, 6 is to move commits to its own electrical structure index.

B

Originally, all our data was in one single index and in the past we have experiments and moving out certain data to its own index. For example, issues. Mrs commons, we have seen improved performance after we split the index by the data type.

B

Now we plan to continue with this effort and we'll be moving commits to its own index. We hope it will help speed up the search for the commits and also help to speed up the search in other datas in the main index.

A

Yeah, so I think I want to drive into this just a little bit, because I think it's really interesting that you know if I'm I'm changing how we index um a type of document, but it's actually going to speed up the performance when I go to recall those documents right. So it's even though it's an index change we're not really trying to speed up how fast we're indexing it. That seems to be okay right, uh it's actually about how you call that information back from the index right right, and so it gets really interesting on.

A

Why that why that occurs,.

B

So we we in our search users can search per document type, for example, issues commits by moving each of the documents type into its own index. It's basically it reduced the index size greatly, so that helped to speed up the search. A lot.

A

Yeah, so it's it's interesting right. This ends up being a part of just how elastic search works right, so we have um an index. Then you have this an index broken out into charts and when a search happens, it actually goes and it hits each of these charts to then pull back. That result set.

B

A

If you have a really large index that throws everything into it, it's got to search across lots and lots of more charts, and so, if we break this down into smaller indices with less shards and that scope is represented to only need to search across those shards, uh the difference is in the performance is seen because you're you're not searching across hundreds of shards you're searching across tens of shards uh to try to build that result set back and uh and ultimately, it makes for smaller shard sizes.

A

So the index that actually contains code starts to see a performance increase as well, because it's actually hit less often than the shard sizes end up being smaller right.

B

Yes, that's correct.

A

That's really great, I mean it's it's it's kind of not intuitive. To think that you know we're going to improve indexing, but it's actually going to perform improve the performance of it. But that's how that's how elastic works.

B

Yeah, I think it's also helped improve the reliability. So if you have everything into one index, if you write into issue with index, basically everything will be affected now by splitting the index by the document type. If you have uh issue, maybe with one index uh the other index can still be functioning.

A

Yeah, that's a that's a good point. I didn't think about this, but you know we also kind of saw this as we did this with issues. Another unintended benefit that we definitely appreciated was, as we started, adding more features and needed to change how something was indexed, calling us to read index it. We didn't have to reprocess everything in the index we just had to reprocess issues, which was extremely small compared to what the code index was uh and instead of spending several hours or maybe even a day trying to do a re-index.

A

uh We were able to re-index issues and I think it was like 15 minutes with the new, the new field that we needed, which is you know it's a it's. Not it wasn't one of our intended advantages, but it does support that there's a lot of other advantages that really come to this, this concept of breaking out that index, uh and so there's a lot of value that we'll we'll continue to see as we we move down the path of breaking out the index into these individual scopes.

B

A

um I think I want to jump to something since we're kind of on that track around a spike that we're taking on too right. Yes,.

B

So this one is to explore other routing and sharding strategy right now our shard is built around the projects. So, basically, all the data is shared by the project id. It provides a very fast search experience for the project level search. However, we also provide the functionality which allows users to search within a group.

B

Now under a group you may have many projects. Those projects can be scattered onto multiple charts.

B

Then, when you search when the user search within a group, the data could come from multiple charts, so the the performance can be in can be affected by aggregating all the search results from multiple channels.

B

Now we are exploring another strategy to shard our data by group. We hope uh this will yield a good result on a group level search.

A

Yeah, so that's you know, it goes right, along with uh how we were discussing breaking out that index right. So this is more into um what determines how you're trying to keep information? That's similar in the same shard and there's lots of ways elastic, tries to balance this out. Yeah.

B

A

The routing key to try to do this, um you know we we force the uh name space. Is that right.

B

A

Yeah as the routing key yes, because it seems logical that a user especially- and you know, self-managed as well as sas, you know if that you're going to be working more times, trying to search across things in the same the same group, and then you know what we're looking at now is, if there's a a better way to speed this up. So uh you know, we don't know right. That's why it's a spike. We want to see if there's a better way.

A

We think there is a better way and uh I really look forward to seeing what the results look like for this and anytime you can improve. You know your document, routing strategy, that's a global impact. It has, you know, far reaching impacts and advantages across every aspect of that you're using uh to call elasticsearch for so it's actually really exciting.

A

If we can come up with a result, either way, but uh definitely exciting, we can come up with a result that that is superior to what we're doing today uh and moving into the idea of what we do for you know in the experimenting round right of how we're trying to find better ways to improve.

A

We have rally right. Yes,.

B

Since we just talked about uh the spy, the spec is about to explore other sharding strategies. uh Now it comes. It also comes to a question how we can determine whether uh the new sharding strategy will be better than existing one. Now uh we need a tool to do the benchmarking. For us, uh we think uh riley uh is a industrial standard tooling, for benchmarking, the elastic search performance will try to build a really testing and benchmarking framework with e-like search at this lab.

A

Yeah- and I think we you know- we touched in and did our first uh internal demonstration on using rally and the um in the last milestone and uh as we were learning more about the values it could create uh it. We were impressed, I think right.

A

Like I mean I was impressed at least I really saw it as you know, a tool that um you know really does simplify how we can go about trying to test comparable versions related to performance testing, and I think, eventually, we'll probably even get into uh ways to improve the quality of the results that come back as well, and you know it seems like it has enough pieces built in that it saves us a lot of time by using it, rather than trying to explore our own methods of of creating those comparable uh testing scenarios, so it at least at this point.

A

We definitely want to keep exploring how we can keep learning down that path and be able to increase the variables that we'll be able to test in a faster method. So I'm pretty excited about where this could take us over the next. You know a few milestones as we really get it into practice and and learn uh faster, the things that it can really help us test.

A

And if we're making all these performance changes right like it, it's also important that we're focused on continuing the quality elements and how we're actually looking at at providing a qa standard uh to ensure a high level of quality, and that's where the next. The next couple issues come from.

B

Yes, one effort we're going to make or we are going to continue making- is to speed up our search engine tests.

B

This is one and we have another one which is to upgrade the version of elasticsearch to a relative latest version in our passing environment.

A

Yeah- and I think this one's actually pretty interesting right because we as we did this, we we kind of discovered uh that the docker hub version of elasticsearch does not necessarily always keep up with the most recent versions, so elasticsearch released uh 7, 15 1 at a time in this video they actually a few days ago. It released 7 15 2., but at least we want to keep up with the latest version available through docker hub, which you know, as of yesterday at least is uh 7 14, 2. and so.

B

A

Why 7 14 2 is the focus here instead of the 7 15 2 that that came out seven days ago? I think yes.

B

Yeah, we will also be looking at maybe alternatives to docker hub.

A

Yes, yeah and that's something that we we hope to include in a a future milestone with some updates, um but at this point we're still talking about what we want to do as an alternative, and uh so for for at least this milestone we'll be focused on. You know, upgrading the version uh to to be the latest available from docker hub, and that takes us through some of the.

A

You know a lot of the engineering effort that we're doing uh coming up now and uh gitlab 14.6, and we really look forward to seeing the ux changes that we're we're looking at as well as we incrementally evolve and make that experience easier and better to use, and we look forward to giving even more feature, updates and and details on our engineering efforts as we go into fourteen seven.

A

B