gRPC Community, 4 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: gRPC September Meetup/ Demo: Connecting Ruby and Elixir microservices in a high-load gaming platform

Description

At Platogo, to enable us to handle high volumes of events from our social game apps, we built out our infrastructure in a cloud-native, microservice way. This means that we can mix different stacks, and have parallel development of services, and have them communicate over gRPC or custom TCP-based protocols. Daniils Petrovs, server team’s backend engineer, will show how we moved out core Ruby functionality into a separate Elixir-based umbrella app using gRPC, how this enabled us to develop faster, and what new changes happened to gRPC in the Erlang/Elixir area.

A

A

My name is Dan Petrov or my full name: Daniel spatrovs uh I am a back-end developer at platoco, and today I have the honor to tell you about connecting Ruby and elixir microservices in a high loads gaming platform with some very interesting details and of course we will also talk about grpc and its current states and the Ruby and elixir ecosystems so without further Ado. Let's start uh I noticed that one of the meeting rooms is called Sound of Music. That's a nice reference to Austria.

A

There you'll see why uh so a little bit about me, I'm, currently, mostly a back-end developer, with some full stack experience, um I've worked on different kinds of microservices, mostly Ireland and elixir, and more recently Ruby as well.

A

um I usually do Elixir developments mostly and nowadays, in my free time, and sometimes even at work. I also do adultments in go and also I contribute to open source projects on GitHub and starts a bunch of new ones, either. Personally, in my own time or at work that Avatar you see. This is something that everyone that's my company gets when they join. So it's kind of immutable and they're supposed to represent uh us as unique characters.

A

It's really nice touch and about my background, originally I started in uh two way and a live casino company. So what is life Casino? In a nutshell? It is uh casino games that are live streamed to people over the internet and they can play them on uh different devices, usually through a web browser or native applications, and it's usually involves real money. Gambling uh add a bit about the company I'm.

A

It's currently the Togo uh we're a pretty small company based in Vienna Austria, that's in Europe, so no kangaroos there, unfortunately, and we have been acquired by this other bigger company called screen tube a few years ago and which is owned by this other larger company called Nova, Matic, uh they're, very big player in the gambling markets.

A

uh We are mostly focused on social gaming. uh What is that? It's is essentially uh implementing all of the social features that surround games and uh platforms which have a lot of users so, for example, XP systems or loyalty system, mechanics or VIP systems missions, leaderboards uh league divisions, that sort of thing and our company is still quite independent. We have less 25 people uh split into three teams, I'm from the back end or because the server team- and we maintain many multiple services in Elixir golang and also one big Ruby models.

A

uh Those are our core parts uh and, of course, our main very new drivers or our apps. Currently they are a South, Park and scheminator.

A

uh The occupy the tough spots in the German and Hungarian Play Store and App Store markets, currently uh with millions of downloads I want to leave the exact numbers, but you can always uh look it up if they show it uh so a little bit of context. What it looks like this is our lobby.

A

uh We to make it very clear: we don't usually make the games themselves, we get them from suppliers, for example, our grounds company is one of them. uh We provide everything around it, so the lobby how the profile system works, how the users can interact with one another, how they can build uh their own teams and complete missions and earn uh in-game currency and experience from this.

A

So all this is using virtual currency. That is just for fun. uh So this is the most popular game. That's our players. Play because from you might have heard about it at some points and the most important mechanic and our platform is uh spins, and that's so when people want to play a game, uh they have a certain balance. You can see it's down there in the bottom left in their accounts, and this determines. You know how much they can actually bet.

A

If they don't have enough money, then they cannot place a bet on a certain game, obviously, and when they perform a spin, they stake this bet, so they either win or lose.

A

This is a very simple version without any of the other mechanics like bonuses, uh boosters Etc, uh just the spins, so uh a spin is kind of like making a role in a loot box and any other popular video game.

A

uh But in our games players do this very, very frequently so uh many many times and minutes, because they obviously want to do a lot of spins and play a lot of games this way, just like in a real Casino, for example.

A

uh So some very basic statistics we have usually around 25 000 concurrent users uh like on an average day during the daytime uh lunchtime, is usually the best time from what we've noticed. For uh obvious reasons, people have some free time to spare when they can play a few games and we usually see about 7 000 spins per second, that's our most important event and our most frequent one, of course.

A

So, a little bit about our platform, uh like I, said we have this one Ruby monolith, which is the oldest and most important uh part of our system. uh It has over a million lines of code and around that we have a bunch of Elixir microservices. These are the newer ones and almost all new services that we develop are in Elixir.

A

We have a few small go microservices, usually for data processing and things like that, or some very small supporting things, and this is all running now on kubernetes on eks uh we have uh traffic and Linker D for our service mesh.

A

uh This is a pretty new thing. It didn't exist, I think, two years back, uh so we're still getting used to it, but so far we're quite happy with setup.

A

You'll see why we have Linker D in the minutes and we have multiple cdns, multiple databases and we have Kafka and Robert mq as our data Brokers, very, very large platform, so why we started moving wallets out. So what is this wallet like I mentioned before every player?

A

They have some kind of uh balance like in any game that has a virtual currency, and it's and making sure that you keep track of the right balance at all times is obviously very important in any game centered around currency, and we noticed that uh this logic was a bit too much for our large remodelers and since we integrate with our Prime company- and we want to integrate with the future Partners as well, uh we needed some more advanced features of Elixir for this, and this is where we realized.

A

We needed to move this out into a completely separate service and uh also at the time I.

A

Wasn't there at the company yet, but at the time uh grpc was starting to gain some traction uh uh and also libraries for working with it for Ruby and elixir uh were there, and they were also fast enough and benchmarks forever kinds of loads, and also we introduced anchor D, which has this really amazing feature of grpc load, bouncing almost out of the box and also a support for tracing, and so that is kind of a very important piece, especially when you're running on kubernetes, and you have multiple nodes for your app and you want to do load balancing.

A

uh So how did we do this? So? The first thing is obviously to set up the repositories and figure out how the services are actually going to talk to one another, and in our use case our monolith would be the clients or the stub and grpc terminology, and our wallet service would be the actual grpc server and for our riskies, just unorange. Grpc is completely sufficient.

A

uh And we decided to write all of our protocol definitions in the wallet repository and then our monoliths can just uh read them and uh create the necessary Ruby classes from there. But you can also just keep a dedicated repository for your photo above definitions. That's a technical detail and whatever works best for you.

A

So on the Ruby side uh it was quite smooth, especially since Ruby is officially supported, since it's quite a mainstream language compared to elixir. So here we have one of the most common examples. Of what we need to do is get a balance of a player's wallet by its ID.

A

uh You just make some sort of convenience method around it and we just call our generated stop with our defined, but that's yet World Balance. We create this new bounce request and we execute it. And then we read the balance from the response so very typical stuff and it can be very nice and make a convenience freak task. uh We made one called generate grpc and under the hood.

A

It just calls further C and cleans cleans up some things and makes makes sure that we can very nicely generate the Ruby classes whenever we update our protocol buffer definitions and on the Elixir side. uh It wasn't that great originally uh so before it went into the main Elixir grpc projects was kind of abandoned for about two years and we had our own work of it.

A

Where we had to do some small patches, the rest of the community also had their own forks with their own patches, but thankfully, in 2022 it's got its first release in almost two years, which is great and now there is very active developments going on in the projects again and if you look at the list of supported features pretty much. Almost everything is there, except for grpc reflection, uh but that's mostly a nice to have it's not that critical and they seem to be already working on it and also this week.

A

I noticed that Jose valim, the creator of Elixir, has some interest in the projects and he made some small contributions there, as well uh in terms of working with proto-c uh uh awkward. Electric actually made a really nice library that adds a next task. The mix mix is the build tool for elixir where you just Define your configuration, how you're going to generate your Elixir modules, and it does almost everything for you.

A

uh So in the context of Elixir, you don't generate classes, it's not an object or is its programming language, it's functional, so uh all of the codes in Elixir has to exist in Elixir modules and modules can be in Elixir files.

A

So that's what he generates so here we have a small example uh where we can get a balance of a specific wallet. uh This I stole from another project that I'm gonna show you in a few minutes, so it's very very similar almost to the Ruby code, since their syntax is quite familiar. Let's see uh it's still slower than the go implementation, but it's still fast enough for us from the benchmarks. So uh no big issue there. We don't have to switch to go just yet.

A

So how does this all look on a very high level? uh This is emitting a lot of the small details of our platform. The most important thing is the Togo application server. This is our most important Ruby monolith.

A

uh This is what our client apps talk to the most and usually it's over http and, for example, we want to start a new game, as so. The most important thing that we need to do is get the player's balance. Obviously, so they know how much money they have to work with, and we also want to start a game session.

A

uh I should have maybe mentioned that in our wallet's umbrella we have the dedicated Ebola service and we also have another microservice called the game session manager, uh because managing the balance and also managing these game sessions go hand in hand, so it made sense to keep them. You know an umbrella as they share a lot of functionality using an intermediary shirts up it's a very common pattern in Elixir um Elixir, mono repos.

A

Let's see so here our particular application server PES, it's chemical, gets well done and gets uh uh specific player's bounce quite quickly and also start a game session.

A

uh This is what it looks like on the more platform side. So there's our kubernetes cluster. We have our traffic incoming and going through traffic uh that goes to PS, and then we have load balanced requests to multiple parts of our GSM service and our wallet service and that under the hoods talks to our dedicated uh wallets database.

A

So that is a very, very high level view of how it looks like so. In this case, JPC was a really great solution for us. Maybe I should mention uh why we are using grpc in this case and why it's not that common in Elixir, but I'll leave that for later. If we have time it's a little bonus thing uh is that with grpc can also make other clients applications that call your grpc servers.

A

So one thing that was very useful for us is making our own uh game launcher. So this is something that we made to. Let us develop without waiting for the client team to implement you finished clients, so this under the hoods uses grpc channels uh and if we implemented this using Elixir and Phoenix Skyview. Is this really awesome framework for making uh web apps without needing to write almost any JavaScript with real-time updates on the UI?

A

uh And this is how it works under the hood.

A

So without going into too much detail to launch certain games with our partners, uh we need some information from GSM the game, a session manager service, and we also need to make uh HTTP calls to our partner service to actually launch the games with special parameters and to get those we need to make some calls to GSM.

A

Luckily, with Elixir. This is quite easy. uh We used the same owned mix, compile task that I showed you earlier. We use the same exact, protobuf definitions. We generate a step client, but this time it's in Elixir instead of Ruby, and we can wrap that into a gen server and keep reusing the same grpc channel uh if you're not familiar. What is a gen server? Maybe if you've worked with akka you've heard about this, it's basically a stateful actor that you can create and it runs in its own uh Airline virtual machine process.

A

You can send messages and get things back.

A

uh So, with all of this put together, uh we made this really nice launcher in about three days, so you just run it and it makes your PC calls for you under the hood and you have a full-blown web app in the browser.

A

So that was very helpful for us because it saved us weeks of time before the client team was done with and entrance working clients with the new partner apis.

A

So what are the results? uh We started migrating all of our acts of users from the internal Ruby Vault implementation to the New World Service this year and now I'm happy to see that's pretty much. All of them are migrated, so we can always keep track of how many active wallets we have at any point of time uh uh we didn't experience any major PC related bottlenecks.

A

So far, the only issues we had were with wallet itself, but it didn't have anything to do with grpc or The Elixir implementation or the Ruby implementation at all, and it was also quite understandable for new Developers.

A

uh It was very easy to onboard them into the protobuf format and how to work with a privacy compiler and how to update the definitions when we needed to create a new requests and response responses, for example, and what will we do in the future uh in the future? We are already looking at using grpc more to let other Elixir Services talk to PES directly uh because very far down the data processing chain.

A

uh It's not that great and it's not as responsive as like a direct communication between our original and the wallet Service as we'd like uh we were also thinking. Maybe we could use it on the very ends: user client, side, I know that it's possible, but that still needs some more research and, of course we are thinking of making more debugging tools. One idea I had is to make um maybe a nice gaussian life for working both with the ball service directly and not having to go through Elixir or through the Ruby monolith at all.

A

This would be really great for debugging, and one last thing I forgot to mention uh why grpc is not usually the first choice when people have purely hurling or Elixir bass. Clusters is because uh distributed early. The way that Elixir notes can communicate with one another uh they already have. Support for remote procedure calls out of the box uh it's already there, so there usually isn't a reason for uh using you know yet a dedicated protocol, which is also on based on TCP.

A

So, but when you have a mixed stack when you can't really just throw your entire Legacy up in Ruby or Java or any other language away, and you want to integrate new services and a completely different stack on the completely different virtual machine than grpc is a really great option. In my opinion, because as you've seen, it's works even across uh uh Paradigm boundaries, so rubies object, oriented, elixirs, uh almost really functional, and it's still works uh quite well together and it's easy to wrap your head around.

A

So that's it for me and in Austria we say: Danke Schoen I'll leave some links for you here. In the end, you can always check me out on Twitter or GitHub or on my website. You can check our websites it'll, be updated, very soon, I promise uh and, of course, some links to the projects that I mentioned.

A

So that's about it for me, thank you for listening.