Grafana Community, 27 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Grafana OnCall Community Call - 2022-07-27

Description

Join our next community call: https://github.com/grafana/oncall/discussions/categories/community-calls

0:00 Into
1:25 Update: Faster Alert Group List
7:06 Migration from PagerDuty
22:38 Open discussion, Q&A.

A

Hello, everyone, uh that's our second community call uh for profane on call product, and today we have uh such an agenda at the beginning, a quick review of latest updates after this vadim will show how migrator from pager duty to on-call works.

A

I guess there will be a demo, uh I think we'll tell more, and after this uh we have time for open discussion, q, a uh most of from call engineers here and we'll be happy to answer your questions. uh We could discuss open issues.

A

Plans ask like questions if you want to contribute, feel free to raise the topic and discuss it. uh So, uh let's talk about updates. First, we have, since the last call, we released a lot a lot of uh releases minor releases, so we didn't change the major version yet and I think that the biggest update which we have it's one which landed by vadim recently, this pro request, which was recently merged.

A

Optimization of alert group list view, and I didn't they want to say a few words about it.

B

Yeah sure so we had an issue with the instances where we have a lot of alerts and teller groups, and we had a huge system on caching, uh so in order to actually show them faster, but now we decided to move out from it because it was like suboptimal and now we don't use caching for now the database level.

B

Caching- and we just optimized some sql queries, optimized our requests, and now it should work much more faster, for example, for instances with half a million of open alert groups, uh things could get very slow like uh 30 seconds, probably or or even more, and now it's not it's not an issue because of this optimization. I don't actually have any instances at the moment to show you, but the increase in performance is very large. We also planning to use some lightweight caching solution to make it more faster, more fast and uh yeah.

B

That's it that's mostly um internal uh things in our back end that allow for faster, renders yeah and now you see our production environment and it got faster and it also doesn't use any caches. It doesn't invoke any tasks that interfere with other ones. So it's uh it's. Definitely it's when they're demoing, it's not very fast, but yeah.

B

C

B

A

Yeah, thank you. It was like 30 40 seconds for some environments, and uh this is about those requests right.

B

A

So they're taking uh 300 milliseconds uh still some field for optimization, but it's much much faster than before. uh Yeah it's landed in version number 13 uh by the way, a few words about how to update versions. uh It depends on how do you use which environment in which environment you installed on call?

A

We have instruction here.

A

A

Here in open source.

A

C

A

That's the starting point for how to update your environment.

A

If you have hobby.

A

Just update your docker image, it should automatically migrate.

A

And in production it it's it's more steps: uh how how hard is it to update uh the uh on call in hell and now? Okay, probably that knows.

C

Yeah, it's basically like that. We just update the repo and then run the helm upgrade command. It should do everything for you automatically.

A

Yes seems seems pretty straightforward. uh Also, one update is that uh newest version of phone call. It is able to check if your front end and backend are the same version. So here you'll see when they're saying version mismatch something like this. It will mean that your front-end version is different from backhand, so it's better to update both just to make sure that versions are matching.

A

Okay, I guess that's the major uh update which will be invisible for users uh since last minute call also we closed a little bugs and we were focusing on deployment issues, uh issues with deployment tech, so, as you see, most of them are distributed between team members, which means that they should be closed soon.

A

So we are focusing on how easy it is to deploy it, maintain and run it in production. uh I guess on this stage of product, it's important. uh Okay. Next uh next agenda topic is.

A

Introducing right so again, let's go and.

B

Yeah uh we have a migrator that allows you to uh migrate your configuration from pagerduty to on-call. So, if you just want to try out things in on call or if you want to have some production environment there check out the migrator, so it can save you time and actually do everything automatically the configuration part. Let me actually share my screen and do a quick demo. So in the on-call repo, you can just scroll down to the in the readme file.

B

You can see the migrator link here and it's actually adopted a python script in docker container, but it's fairly easy to use. You only need your page reduce api token, on-call api, url and on-call api token. So these are, I will show you what these are and how to actually do it. So let's go uh over the readme.

B

So currently, this tool allows to uh migrate your user unification rules, your escalation policies, your on-call schedules and also integrations as as known as services integer duty. So um let me actually show what I mean by this. So the services they will be migrated. For example, if I have service one and also have, for example, promote use integration for it, it can be migrated also, it will migrate on call schedules.

B

So I created some small demo environment pleasure duty when, where I have one schedule and I'm on call for it, so the services, the on-call schedules and also it can migrate your user notification rules. So in this case for me, I only have email me, so that's pretty much it and the next. I guess escalation policies yeah.

B

We forgot about skeleton policies, so if you click escalation policies and see some of them so, for example, for escalation policy, one we have 95 volumes for me, basically in and then after 30 minutes notify uh the schedule number one and schedule number one as well, I'm on call for for for this schedule.

B

So uh these are the resources that you can migrate with this tool. It has some limitations.

B

For example, we don't support some types of integrations. We also migrate badger duty calendars using the icon link. So if you migrate it you will need to actually recreate the on-call schedules manually.

B

If you want to, and also we have some some differences in our system, so we try to match your configuration as close as possible, so there will be some slightly different from original resources so that that's it on the what it is and limitations, then we, I will show small demo how to actually run it, so um you will need to have docker on your uh computer. You also need to build the build the image locally. You will need to get your api tokens in pager duty and final on call.

B

So uh this part is really straightforward. You just need to git clone and go to this directory and build it. I I I've done it myself, uh like five minutes before the call, so I will not spend time on this and then also you will need to go to get your access keys so, for example, in pagerduty you can do it.

B

I guess in in your user, for I will do it like this in user settings, you can acquire an api key, so I already acquire the key and add it to my nf and also in your here's microphone and call it's not fresh. So it has some some integrations some users, some escalation chains, but this migrator tool actually will not delete those and only add the resources that are in major duty, but not in on call.

B

So uh in the encode you'll need to go to settings, and here you will need to create an api api token and also get this api url. So, basically, uh when it comes to running the container, you will need to specify your page reduce api token on call api, url and call api token.

B

So, let's, uh let's pretend that I built the image I acquired both api tokens on pagerduty and graphene on call, and then I want to actually there's two uh mods of running this migrator. You can either plan or migrate. So if you want to just to see what will be migrated, you can use it in plan mode. So let me just.

B

Run plan in my terminal, so currently I'm I have a 10 file, so I will be not specifying the tokens because, like it's, it's not safe. So I will use my environment for this, but you will need to specify it like here.

B

So let's run it in the plan mode and see what it does. So, basically it will fetch users, fetch schedules, escalation policies and integrations both for on-call and both for pagerduty and see what what should be migrated in this case uh everything is red and you can actually see in the error message that there's uh an error, so the graphanol user, no referendum called user fund with this email.

B

So let's see the next step to fix that I want to go to my users page and see wow. I used my personal email here, so I want to actually go and fix this. Let me.

B

So this user will be so I use my personal email or, for example, you can add a user with with this email, so I will be using microfan email.

B

And you go to encore just to verify that.

B

Everything is working so yeah, you see the my email and, let's rerun it again, so we plan the the migrator.

B

Okay, wow a lot of green stuff. I like it. It means that so uh use notification. Rules will be migrated for for this user for user with this email. Also, now, if you see here, it can't just go and migrate the schedule, because some users in the schedule are not present in grifanon call.

B

But now, when I fix this fix my user email, I can actually go and migrate the schedule it and it will work as as expected. So the same goes for escalation policies, because in some escalation policies reference it referenced uh this user that was not existing in graph and uncode, and now it's fixed. The same goes for integrations.

B

What you see here. The red point is that uh for, for example, service two which was pingdom. It couldn't be migrated because we don't have that type of uh integration refinance. So it will tell you what it will be. What will be migrated? And let's actually at this point, I'm happy with the report.

B

So let's just go ahead and migrate. It.

B

We run it in migrate mode.

B

B

Run it in migrate mode.

B

So it will migrate, user notification rules, schedules, escalation policies and integrations only integration that it would migrate because this one it's we can. We can do that and now, let's see, let's just refresh the page, so use notification policies, let's see for this user, for it has this sms. So what happened? We had this uh notification rules. We had this email, but now it's sms. That's because currently on-call doesn't support the emails, the email notification type and we matched it to sms. Just to make sure that you receive something in in some type of notification.

B

That's basically it on user notifications. Now we can see the next resource, which is schedule. So, as you remember, there was a schedule.

B

Which was called schedule one and I'm on call for like for a long time, uh and you can see that we actually have this schedule with this user and the dates are uh correct, so I'm actually on call now for this schedule to enumco.

B

Then we have our escalation policies, uh for example escalation policy, one for service one, so it will ping me wait for 30 minutes, uh notify the schedule, and then it will uh wait for 30 minutes. That's basically the same. That was in a pager duty when we have escalation policy, notifying the user and then waiting for 30 minutes. Let's find the schedule, so that's pretty much it and the integrations part of this, uh for example, service one that was prometheus.

B

It automatically connected this escalation policy and the use this is creation policy which will do the same. So if I actually press the same demo alert, it will actually.

B

It will send an sms to me and yeah, as, as you can see, you can easily configure your on-call instance just using your pagerdutia account and also all the schedules, all the installation policies will be intact and you can just try out your configuration in pagerduty trying to migrate it to on call and just trying things out. So you don't have this like blank pages.

B

Also, you can see in this majority migrator readme page the different types of errors different, but basically I covered all of them and yeah the the last thing after after migration, we would need to reconnect the external monitoring tools, so it pointed to pagerduty. It will now point to graphene uncle and also we would need to tell all the users to connect their phone numbers. Like accounts, telegram, accounts and steams accounts just so they get their notifications, and the tricky part is that at this point we use the ical file from pagerduty.

B

So if you want to migrate fully into on-call you, you can just create a new schedule uh using our built-in google calendar, uh google calendar integration or anything like that or tier form. So at this point, if I create it here in pagerduty, it will be also reflected in encode yeah.

B

So please check out this uh small little directory. It has some bugs, but I'm not just. I will open up here and push it soon, but yeah it it. It works. Well, it won't mess up your configuration. It will tell you exactly what it will do and what it won't do. So please check this out. uh Try refund on call and see if it works for you so yeah, that's pretty much it for the pagerduty integration of patriotic migration, so yeah back to your monthly.

A

Thanks again, I have a question yeah, my so so. uh Actually what does it mean uh when uh we migrate from pagerducey to oncol? uh Uncle schedules are still kinda like connected right. So uh if somebody wants to try a phone call, they could just they they could not like kill pager duty. They could still edit uncle schedules and pager duty and check how on-call works.

A

So this this side effect is uh available. If you want to trial on call- and you want to make decision about- switching based on like if you like it or if you don't like it.

B

Yeah sure you can use both systems intact. Just add another receiver, for example, to permit use or add one receiver to one call also use the same schedule that you can edit via pagerduty.

B

So you will be good and you will have two monitoring systems and just you can uh try it out and don't break anything in case uh in case you want to migrate.

A

But yeah, if, if you will change here, for example, user notifications, which are copied which are not linked, uh they will unsync with pages.

B

Yeah, you could, you could actually run this script like infinite amount of times. It's it's important, so you can like if you change something just rerun it and it will create the same stuff.

A

Oh cool, it didn't understand it, you could run it.

A

uh Any more questions about this.

A

Yeah, let's see claps, uh okay, that's that's all we have in prepared agenda now free some free time to maybe if somebody has questions or proposals or something like that feel free to speak up.

A

How so, how should we build the object exporter.

A

uh What's the schedule for integrating email intel escalations.

A

Let me let me tell a little bit more about how do we reflect our plans in public repo.

A

So if you want to see what we will assets internal team, of course, anyone from outside could work on any issues, uh especially markets, good first issues and thanks to those contributors who already participated and proposed for requests, we already merged a few. uh If you want to see uh what we have as a team in our inside backlog, you could see the check the signage, so we once we transfer issue to a mid like short-term backlog. We assign team members to them.

A

This particular issue is about email.

A

No, we don't have it.

A

Wow, that's replaced.

A

uh Okay exchange, I I was sure we have a mission about in my own situations. uh Actually, it's planned for our quarter next quarter, so next three months uh we should be able to deliver it uh also. I know uh mathias already prepared to request for smtp, based in my notifications, right.

D

Yeah, that's right. There is a full request, implementing a really simple messaging backend that you could pull to your own prem installation and configure to use smt n70p credentials, and it will send you notifications, using email. There is a pr with the instructions how to set it up, so you have it there. We go.

A

Yeah this one really.

A

Yeah it will require uh to build dockercon container. Of course,.

D

Right, you will need, depending on your setup, you will need to rebuild the service.

A

Yeah, I think I think we could merge it pretty soon. So uh this particular feature, I guess it's some short term, maybe a month like within the month, we should be able to merge it test and merge, and uh in three months we will be able to show something really robust production usage or maybe just test this thing more and roll it out.

A

Oh you're on graphing cloud yeah, so within three months, which we should roll out uh email notifications to graphing cloud and open source. So it should be everywhere.

A

That's the uni unique moment when open source has more features than a file called version. It's usually not like that, but if you reverse this, like so you'll have more advanced, open search version.

A

A

Or we could just wrap up three two one: okay, thanks. Everyone.

C

Thank you. Everybody bye, bye, bye,.

D

Bye, bye, bye, everyone.