GitLab Deep Dives, 7 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive on Elasticsearch-rails Integration

Description

Subtitle is provided for clarity.

This presentation will briefly introduce the elasticsearch-rails gem, then walk through how GitLab uses it in a slightly different manner.

---------------

Read more about our product vision: http://bit.ly/2IyXDOX
Learn about FOSS & GitLab: http://bit.ly/2KegFjx
Get in touch with Sales: http://bit.ly/2IygR7z

A

So hello, everyone, my name, is mark and I'm currently in the fulfillment team, but I was in the global search team a while back and I implemented some of the stuff. So I thought I should give a presentation before my memory fades away.

A

um Today's topic is about elasticsearch for those of you who don't know how elasticsearch is it basically offers a global vortex search. uh You can do a global search on the top right corner. The following search field.

A

There's no demo for today, because all the changes I'm discussing are, they can changes and they wouldn't affect our users. So uh most of the slide will be about ruby codes except the very very end. So it's so more focused more for the rails developers.

A

Today's session will be split in two parts. The first part I will be discussing how the elasticsearch rails gem works.

A

Then I will discuss how gleb uses it and why we use it differently.

A

Since the word elasticsearch is quite long and sometimes it doesn't fit the screen, so I would abbreviate it from time to time so part. One elasticsearch rails is a rubygem maintained by the elasticsearch company. It contains three parts. The first part is called elasticsearch persistence.

A

It basically allows you to use the elasticsearch server to store all the model data like mysql or postgresql. We don't use this gem. The second part is elasticsearch rails. It provides many utility functions and we use it for instrumentation.

A

The most important part is the elasticsearch model jam. It basically connects our rails model to the server, and this is the only part I will be discussing today.

A

So elasticsearch models course to provide a proxy to bridge rails model and the server all the search related logic resides inside the proxy and all the search commands will go through a proxy to the server.

A

There are currently there are two types of proxies: one is class method proxy and the other one is instance method proxy, as you can tell from there, one is sitting at a class level and the other one is sitting at the instance level.

A

So if I I can access the classmates of proxy from the issue class and I can access the instance method proxy from one issue object.

A

The class level proxies are mainly for reads: for example, when I do a search you will go through the class message proxy and the instance method proxy will be for rights say. If I create a new document, then I want to update the data to the server.

A

The most basic setup is to just include the two modules: the elastic search model and the elasticsearch model callbacks. By doing so, we gain all of search functionalities.

A

For example, we'll get two elastic search methods to underscore elasticsearch methods. uh What this method gives us is. It will return the proxy we mentioned earlier, so for, for example, a book instance. If we call underscore elasticsearch, we get an instance method proxy.

A

Its target will be the book itself and if we call underscore elasticsearch on the book, we will get a class method proxy.

A

So the proxy object is like where all the search related logic resides in. Basically, we can call all the commands on the search on the proxy object.

A

Here I will use the indexing, as example, to show how the gem works. So if we create the issue- and we save the issue, what would happen.

A

Since we included localback module earlier, the model module will add three copics to our model, one for create one for update and one for destroy, uh since we are in creating an object today. So this uh the first line will be triggered. It will call elasticsearch proxies index index document method.

A

The index document method will first prepare the data for indexing. So if you look at the point, 1 we'll see equals as index json method.

A

By default, a synthetic system, index.json method would serialize all the attributes of the hash.

A

And then this this hash will be passed to the server by the client. As we see on 0.3.

A

By this time, by this point, um the operation is down, so the data is indexed.

A

But sometimes we don't really want to index everything we might not want to make every attribute searchable.

A

We can define our own synthesization method to override the default method, so we can define as index a syntax json on issue model, and here I just want to index a title, because I only want the title to be searchable.

A

The method underscore elasticsearch is very long, so it's quite tedious to type that all the time so the elastigen elasticsearch rails gen, provides some convenience forwarding.

A

It will forward a few methods to learn proxy directly, so we don't have, for example, when we do import, we don't have to type underscore.

A

Elasticsearch uh proxies are consistent of many modules. This means we can cherry pick only the things we need.

A

The design philosophy is very modular, which is a good thing, because we it gives us a lot of flexibility to customize. It.

A

And each of the proxy has access to a client. The client is the object we use to communicate with the server. It's basically a faraday client hour a day, client, and it knows the information such as the server url and the port right before all of the proxy share the same client, but this can be changed as well.

A

So as a summary loss, most simple setup would give us two methods, one on the class level and one only instance, level, and that will give us the two different proxies to work with.

A

And in gillette we use elasticsearch rails a little differently. This is because um first I will talk about our problem. So up until I think last year, elasticsearch indexing can take a very long time up to many days and every time we do a schema change. We require indexing.

A

However, during re-indexing the search results will be incomplete. So let's know yes, so our goal is to have zero downtime search when our data schema changes.

A

The development path was decided to allow multiple versions of search code to code exist at the same time and we determine which version to code at the wrong time. So we have multiple copies of the logic. They can be slightly different and we choose dynamic click which version to use.

A

So I will explain the implementation in three different parts.

A

First, I'll talk about the proxy part.

A

Here we use snippers example. Previously we used to have snippets include snippet search module, which contains all the search related logic.

A

However, the module does not allow dynamic, swapping so less less flexible if we want to have multiple versions. On the other hand, using the classes and objects is a very suit.

A

It's a suitable way to do dynamic version.

A

Changes, the proxy design is flexible, that we can use separate classes instead of using the same class in all kinds of search. We can subclass the proxies I introduced earlier.

A

For example, we can have one sniper class proxy and one sniper instance proxy.

A

And we can also have different versions of the proxy 2., so here we can have version 12.1, sniper, class, proxy and version 13.0 sniping class proxy. We can also keep things dry by extracting common logic here. The common search logic is extracted as application class proxy, and you can see the inheritance hierarchy from snippet to application class proxy, then to class message proxy.

A

So how do we choose, which version to use we use a switchboard.

A

So if we recall earlier the simplest setup, the underscore elasticsearch method would lead us to the proxies.

A

Now we can have the underscore elasticsearch method lead us to a switchboard, which in turn, would link to different versions of the proxies.

A

At the time of when I was writing local, I didn't have a grenade in my head, so I nailed the switchboard classes as multi-version class, proxy and multiversion instance proxy.

A

They are not great names.

A

How do we choose? uh Which version to route to the answer will be? This is a base case by case, for example, if we have two indices version, one and version 2.

A

Assuming version 1 is in sync and version, 2 is still indexing.

A

If you want to do search, we obviously want to do that on version 1, because that's the synced version.

A

But if you want to write to indexes, we would want to have the data right written to both version, 1 and version 2., and if you want to remove the whole index altogether, we obviously want to manually select which one we want to remove the index.

A

So I've pulled us into this table and how do we represent version? One here indicated on the red cell, so we, if for reads we want to return only one version which is the synced version, so I have declared a method called elastic reading target which returns the single in-sync version.

A

For elastic writing targets, you will return all applicable versions.

A

Here it will be version 2.1 version from 0.2.

A

And how do we represent the index, all the methods which are right methods? I have declared a method called methods for all right targets, basically this array of methods which will require us to forward to all the versions.

A

So if we index the, if we create a document, if we remove a document, if we update a document, we want the changes to be propagated to both version 1 and version 2., and this method will contain the list of methods for doing this.

A

And for methods such as removing indexes, we also have a list here called methods for one right target so for import, create index or delete index. Those methods are those methods shouldn't be forwarded to older versions. You should be deciding which version those commands are wrongs on, so they are in the separate method.

A

So, to recap earlier the table I gave can be represented as those different methods. I have declared.

A

For read operations, it would be delegated to the elastic reading target method for right targets, which should be delegated to all the versions.

A

I will look into the methods for all right targets to get a list of methods, and then those will be forwarded to all versions, as indicated in elastic elastic writing targets and, lastly, methods for one right target and those are methods which shouldn't be forwarded to versus automatically.

A

So how is the table represented in code? um I have? The table can be is represented by the generate forwarding method.

A

It is responsible for set up method forwarding at put time.

A

For each of the method, in methods for all right targets, we will call forward to all right targets. This is the point one I've put on the top right corner so for all right targets, sorry for methods which should go to all the brochures. We would forward calls to all versions by calling the forward to all right targets, and if we look in silo forward to all right targets method, you will see that we define a method for each of the methods we pass in.

A

First, we will look through all of writing targets and we will just pass the data by calling public send method to the proxy.

A

Lastly, we collect all the responses and we return unsuccessful ones if it exists. But if everything, if all the version returns successful results, then we return the last one because they are all successful.

A

This is for writing to multiple right version right targets.

A

Now we move on to forward into one single read version for read operations. We only want to forward to the one version which is in sync, but how do we determine which methods should be forwarded to the read target?

A

We will first take all the methods and filter out take out those which are for right for write operations, as indicated in 0.2.

A

We also take out those methods which are the instance method of the current class. And lastly, we don't want to forward method missing.

A

Now we have read methods for all the things we want to forward to the single in-sync version, and we call forward read method, which is a simple delegation code.

A

I'd like to take a detour here and talk about real class, so elasticsearch tries to be smart and overwrites the class method, on instance, proxy.

A

So when I call class on snippet instance proxy, it would give me snippet class proxy, but this can result in a really cryptic errors, and sometimes I really need to get a real class.

A

So to do that, I declare my own class called car. I declare my own method for real class.

A

And you can see real class in use.

A

Here, because I need to get a real class in order to get all the public instance methods.

A

So that's why realcars exists.

A

So how are the targets specified?

A

uh I have a mixer conversion, but if I pass in a version stream, you will give me the class of that version.

A

First, I would get learning space of liberation and then I will get the proxy nail by concatenating the string. For example, if the data is snipped, the proxy class name would return snippet instance proxy, and then I will use this string to look up the namespace to get the proxy class.

A

Let's all for the switchboard, and now we are in the final part, where we want to connect rails model to the switchboard. Each of the searchable model would include application version search.

A

It provides elasticsearch methods which returns the switchboard class and also the sound permission, checks and some other stuff.

A

By including the application version search module once again, we get two elastic search methods, but this time you will return the switchboard objects, one for the instance level and one for the class.

A

Level, so this is the overview graph where all each of the classes I have talked about on the top left corner. We see our rails model and in the middle, are the two switchboard objects and on the right are different versions of the proxies.

A

And for reads, you would go to the single instinct version, but for rice you will write to all of the available proxies.

A

Now we have two different versions of the search code and they can have their own client and their client can point to different indices, only lossy search server, so they can act independently.

A

And theoretically we can even point the client to two different clouds. They would enable us to migrate from one cloud to another.

A

Currently, we only have one version and we hard code that version to elasticsearch reading target elastic right targets.

A

This is because, halfway through, we switch our focus to to enable global search on gleb.com, but I still think there are some benefits. For example, we get, we move all the search related code into their own classes, so our model is cleaner and smaller, and now the testing can be done on the proxy objects.

A

So what will be the next step? Maybe we can rename the switchboard class to actually be named a switchboard, or maybe we don't need the switchboard, because currently we only have one single version.

A

Also, when I was writing this presentation, I find out that when we call generate forwarding it's done many times, I think this can be effective to the only code once during the boot time.

A

So, let's all of my presentation- and I want to give special thanks to marcus marcel, dennis cava, dava and kai- we work together when I was implementing this logic and thanks james lopez, my manager for allocating me time for writing this deep dive session.

A

So it's q, a do. Anyone have any questions.

A

No okay cool and thank you for attending this and wish you have a nice day see you thanks.