GitLab Configure Stage, 9 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitLab Agent for Kubernetes: leader election MR overview

Description

https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/716

A

Hi this is mikhail. um This is an overview and the demo of the leader election merge, requests.

B

But first some background.

A

So the functionality in the agent is uh split into modules, so that the program is not a big monolith, but has some internal structure modules are well? Each module quite often is split into server and agent side component, so there are basically server modules and agent modules.

A

Both server and agent modules have individual apis that they can call. This is a.

B

This is the oh wait. This is the server api.

A

Which is the same as the common api, then very interesting as.

B

A

And then there is configuration that this in this case the server module is given some configuration and the agent model is given some configuration.

B

B

So the problem is that.

A

Sometimes, depending on configuration modules may need.

B

A

Only run as a single tone, so we have active and passive.

A

That's how I call them here: uh module some of them are active, like githubs, contains security scanning and then more here like home support in the future, the only one uh we only want, at most one um instance of aging k, pod to be acting on the configuration for efficiency and other reasons, and there are some other modules that are always on basically so for this modules for active modules, you want to do lead relation basically, so that only one is active at the same time, but we only want to run leader election if the module is active, so is configured to perform.

A

B

Perform some functionality.

A

And if configuration changes which the user can do dynamically using a configuration file, then the module should say I don't want to be: don't want to participate and need their election if, if I'm not interested or on the opposite, if this is enabled, then the module may may signal that it needs to be executed.

B

If, if configuration is enabled, so this is quite a tricky problem,.

A

Because there are multiple kind of moving parts, dynamic configuration and plus multiple modules with such dynamic configuration where each may say I am enabled I'm disabled and if at at least one is enabled, then we need to run leader election. If none are enabled, then we don't need to run leader election plus.

A

I really wanted to optimize for simplicity so that when I'm building a module, I get a simple api, a wait for a module to say I'm interested in this functionality or I'm not interested. Then you know the.

B

Module should just be what it is today without any.

A

Implications- and it looks like this- this is how it looks today. I a module- needs to implement these three functions, this interface and it's fed with configuration and like dynamically.

A

So when the user changes configuration module, gets a value.

B

From the channel gets that configuration basically so.

A

This is the interface I came up with it's a leader module, so one extra method. This has the same signature. I just wanted to add extra docs here. That's why I added it as uh here as well, and from that perspective I think this looks really clean. So we have this gitobs agent module and this is the diff. Basically, if this function returns true, that means this module should be ran for a particular configuration.

A

So you know, as you see, if there are manifest projects and configuration, then this module should be run. If this agent k instance uh pod, wins the leader election and same with the star board vulnerability. If it's configured, if the module is configured, then it wants to participate in.

B

In the election, okay.

A

But yeah this looks really clean, but the implementation.

B

Is is quite quite tricky and that's what I wanted to show.

B

So first we wrap wrap modules here.

A

So if, if this module implements leader module, then we want to wrap it in a special wrapper that.

B

B

B

Wrapper is here, of course,.

A

Yes, so if the module wants to to run for this configuration, then.

B

A

We we push it to this runner and then, if the runner runs.

B

Runs the election uh somewhere here yeah. This is.

A

F4, we select quite an interesting pattern. Basically, this is a run. This is an actor pattern from.

B

What's the language code.

B

I forgot anyway so that basically.

A

This this one uses the same pattern for with the select. So it's like, I think, of of this thing or this pattern as um like imagine a separate process, a separate program that runs and processes, incoming and outgoing signals basically and it holds some state. So in this case we have some state as fields as variables and and fields, and we get incoming configuration.

A

We do things and then we send configuration to another actor basically, and then we get signals from that other actor here as well, and this is that other actor so also think of this as a different like program that acts on signals time to stop time to add a function that wants to run as a lead as a leader and uh time to stop and start.

A

We started leading and we stopped leading like it's a thing that dynamically processes, incoming signals and dispatches signals to other actors, and here we have uh automatas on based on status of the the.

B

Of that actor, basically.

A

So actors plus vinita automata. Basically these are two ideas that I used here.

A

Yeah, based on current state and incoming signals, we make decisions about what we do next.

B

A

Anyway, you can dig into this code, and I tried to comment, leave a lot of comments. uh Yeah.

B

On your own and.

A

Hopefully that is not like it was difficult, but I think the clean api is worth it anyway. I wanted to demo this.

A

So I have two configurations: rancher line running locally, and this runs against staging. You can see that is staging, and this.

B

Is the first one, let's run the second one I have it here.

A

Golan likes to compile things every time.

B

For no reason and okay, we have to run.

A

um Oh one important thing is they use different pod names so as if they are this one is the second one, and this one was the first one. So I as the as if they are different pods, because spot name is used for identity.

A

So each one is a separate thing and then they understand who is the leader now, because they have different names and they try to.

B

They are trying to grab the lock using that particular name. So, okay, we have both running.

A

B

Then, let's change the configuration actually, let's check, let's check the running. Yes, this is the super agent.

B

B

B

B

This will also.

A

Kick off a ci job but that shouldn't interfere. But we may see some logs.

B

From the ci tunnel.

A

So I have enabled gitobs it was disabled before.

B

As you saw and.

A

Now, okay, attempt to acquire lead release, that's a new message.

B

A

Attempted and succeeded.

B

And this was 16 44 41 and this is 16 44 43. So this one won.

A

By two seconds.

B

A

And then it ran github's applied the resources, and this one did not.

B

Okay, let's stop this one, and now this one acquired the lock and also.

A

Synchronized, oh now, this is the logs from.

B

Ci tunnel from the side job that lists pods, nothing else: okay,.

B

A

Acquired lease now, let's run the second one again, it should try to grab the lock and.

B

A

Well, it doesn't fail, it keep trying. It keeps trying.

B

uh We just don't see anything.

B

Oh, we can also do rents and see that.

A

The second one became leader and then the first one.

B

Became the leader when we stopped the second one.

A

They don't there's no events for when it stopped being a leader and they send.

B

A pull request to kubernetes libraries to publish such an event, so hopefully we'll get that and.

A

B

Let's disable detox, you see how they react.

B

B

I think we are pulling the server every like 10 seconds, so synchronizing objects and then applying so.

A

It can we cancel it uh yeah. So there's a.

A

There is a race between.

A

Module receiving the stop signal and the configuration that.

B

It gets so it gets configuration first in this case.

B

B

And now we can also look at.

B

This is the lease.

A

Object in the default namespace and we see that no one is the is the.

B

Leader here and we enable it.

B

Someone will become the leader.

B

Yep yeah, so this one must be there yep. This one is two.

B

Yeah agent four block, so it seemed to work.

B

That's the demo. Thank you.