GitLab Support Operations, 27 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Support Ops Sync - 2022-09-27

Description

Support Ops Sync - 2022-09-27

A

Thank you ale. It is the 27th possibly where you're at it could be. The 28th could be the 26th. Maybe it's January I, don't know anymore times, weird.

A

A

All right, uh so the first big thing is I rolled out the change, criticality workflows and my phone apparently back right now to go off.

A

There we go all right, so um I'll post the link in the zoom chat, but it is also a meeting agenda doc. This is a change, we're going to be doing on all our issues, Mrs and all that fun stuff moving forward. So to give you kind of the quick run through of what it's like.

A

There are four change: criticality levels, um one two three and four because stuff not original. um Essentially it is dictating how big an impact the change has, what our communication style are. How far you know how far in advance, we have to schedule this et cetera, Etc.

A

So, um to give you some real quick definitions like uh criticality, one would be like changing the functionality of a very widely used and desk view like if we were going to change the uh all the SSG or the sgg. Sorry, if we're going to change all the sgg views dramatically in a way that would impact workflows, that's change level, one. If we were going to change SLA at all, that's change level, one um altering like the zendesk in some way. That would be a significant workflow process change.

A

um A good example would be kind of some of the stuff we did in the review, where we added new um stages and stuff like that. I would call that a criticality one, it's a pretty big impact to the overall process of housing desk is working.

A

uh Two would be like we're updating the theme, so obviously that's user facing it's agent facing it's everybody facing. So that's like level, two change, um adding a brand new ticket form that is customer facing um changes to triggers or automation, uh Automation and I can't talk today, apparently uh automations that are relating to like ASAT or support kpis. So anything we did that changed how the asset survey went out or how assets processed, or we did something that changed.

A

The definitions of like what frt and nrt is stuff like that, be a change of criticality too um three it'd be like adding a new field on a form or bulk, removing expired Zen desk organizations or adding a new zendesk app. That's going to make things better for.

B

A

um Removing or deactivating macros stuff like that and then criticality four is kind of our there's. No real impact for these, so adding and removing users creating or updating macros like we don't need to advance. We don't need to announce those in advance, they're kind of just things that happen so um scheduling wise, like a criticality one needs four weeks in advance time. uh Four weeks in advance, two needs three weeks: three needs two weeks and then criticality four doesn't need anything. It can just be done immediately.

A

um The link I have for all this also says who can approve an exception, so let's say somebody says we want to change the slas on this immediately. Okay. Well, that's going to require the VP of customer support to give his approval or her approval, depending on the future um for that exception. Without that our response is just going to be. We can't do this sorry.

A

We can schedule it four weeks from now, but we cannot do it tomorrow today, all that fun stuff- um and you know, if an exception's needed paying you the person or persons who can grant that exception and then once they have commented on the issue Mr whatever? What have you granting that exception, then we can consider doing it. uh Just because somebody has granted exception doesn't necessarily mean we can still do it. If somebody was like radically changed in desk tomorrow, I approve everything. That's great.

A

We still can't do it like that, like we don't have bandwidth, nor the ability to do all that and test it and get it deployed in less than 24 hours, um and then communication is something we're changing with this criticality.

A

So for all of our changes, except criticality, four, like four- is the exception to everything, because you just do things immediately, but um for all the other criticalities you're gonna Post in the support Ops channel uh support Ops announcement Channel using that support Ops announcement uh workflow that we have built you're going to cross post and support team chat and you're gonna cross post and support us Federal, but only if it impacts, U.S federal as an example I posted about removing the expired orgs today, I did not post U.S federal because it doesn't impact them.

A

That's only on global, so uh damn that'll mostly come with. You know your territory because usually the other things we do aren't going to impact U.S, federal and then you're going to make a swer item. Support require s a form to fill out to do that. um Pretty much is like copy pasting from the actual announcement itself. So it's not too bad, but um all of what I just said should be done every week leading up to the actual change. So using an example. I've got documented.

A

If you were making a change on December 30th, you should be doing all of that for December, 2nd December, 9th December, 16th and December 23.. So that's giving four weeks of announcements for a criticality one level change um using the same example if it was criticality three you'd post on the 16th and the 23rd and obviously.

B

On the 30th.

A

You do the thing and say: hey: it's done if um I had one thing that was kind of asked was so who communicates to this Channel or other this team, or all that the actual requester is going to be in charge of that. So whoever requested it needs to be the one communicating to other channels. Other teams Etc we're just going to handle core communication, and the reason for that is everything is way too circumstantial for me for any of us to Define it.

A

So the requester needs to make that decision or dri I should say for the requests, but um that's basically what we're going to do on every issue. Every Mr, some of our issues in the Mars will Auto add a change label for you. That is just because by default, that's probably what it is. A good example is the macros are always going to add change for chances. Are it's always a change for?

A

But uh when you go.

B

A

Your issue or Mr, review it and decide if that's correct, if it's missing one add one um and then what we're going to tell people is to subscribe to those labels. So they can be aware of those changes that have the impact they're looking for, like I, will be subscribed to every change level, one so anytime, one is filed, I'll get an email and all that fun stuff, so I can say: okay, that's a change level. One I need to be involved with that.

B

A

B

Yeah and I I have a question regarding that, for example, uh you mentioned in this view I'm just clearing it up. Zendesk views like changing them completely is definitely correct, criticality one but adding just a field in the existing view. It is.

A

Adding a column, yeah I call that criticality three.

B

A

It's debatable between the two, depending on.

A

To the far right you might not be in people's physical view anymore. That might not be that that's something we would want to announce, but something like we're, adding this column at the far right of the page, all right, so it's gonna.

B

A

Same impact as if yeah, so that's, why uh there's not? Some of this isn't like super hard to find because you're gonna have to make some judgment calls on it. So.

B

Something like.

A

Adding a column not a big deal, because you know it's.

B

A

Have an impact, but it is not significant, whereas like oh we're, removing eight columns and adding all these new columns and changing the how the conditions are and stuff like that. Like that's a dramatic change to that view and that's something that needs to be done well in advance.

A

So yeah, like I, said like some of this you're gonna, have to kind of use. Judgment calls on this because if I tried to make it, you know if we try to make a definite list, you're always going to have. But what about this? But what about this? But what about this?

A

And the answer is: there's always edge cases, exceptions, Etc, I, trust, all the people in the support operations team to make that judgment call if they're ever unsure, you can ping me or Lyle, or you know you appeal as a senior any of those people. You know ping. Anybody else feel like I'm, really not sure on this, and they can kind of help. You look it over and say, look I think it's! This I feel like this is what we should call it.

A

um Any other questions comments, concerns about the criticality stuff.

A

Cool, so just moving forward as you work, your issues and Mrs. Add that label there if one's already there just review to make sure it makes sense. Like I said all the macros are going to Auto, add four but deactivating or removing a bunch of macros is criticality three, so you might, you would have to change the label in that situation.

A

Although I might review that because I feel like deactivating, macros that haven't been used, I mean I might change that to removing or deactivating active macros, because inactive macros, don't matter but like I said judgment, call Okay cool. So the other thing I wanted to bring up is to align with data retention policies, and you know the law we're going to start deleting orgs that have expired.

A

Three plus years ago, starting on the 3rd of October I've been working on the sync script to handle all that for us, which is awesome, it'll make us not have any workload and it'll all be automated, and we won't have to worry about it.

A

The caveat is that first run has about 2600 orgs that it's slated to delete and while the script can handle that I really want us to review those 2600 orgs before we actually just bulk delete a ton of them as such I've shared your Google Sheets and I've put it in our doc, but also posted in assume we're here.

A

um If you look at this, there are four tabs one for me, one for the meal, one for Dan one for Dylan and on each one. It has an org name, the Salesforce ID, the Salesforce short ID and a status column.

A

Your task is to review that org go into zendesk, find them look at them and see if they have any active tickets. If all their tickets are closed, we're fine, you can mark them. You can change the status to uh good to delete. That's that's all! You have to do on that one. If they have active tickets, Market active ticket do not delete now. Your objective from there should be to try to get those tickets closed, so reach out to the assignee or a support manager and be like hey look. This is an org.

A

That's been expired for a long time, they're slated to be deleted. They have an open ticket. You really need to get that close.

A

um I would also say, review that requester and see if they're in the correct org, they could be assigned to a very expired org and that's incorrect and we want to get that fixed. If you get fixed well, then that word doesn't have any more active tickets that you can mark them good to delete if, when reviewing you're, really Unsure, how to proceed, take your o put it on my tab and just remove it from yours and I'll review it and make judgment calls.

A

The objective here is just to make sure that when I do this, when we do this bulk deletion and as you can see, it's like 664 orange per person, I really want to make sure we're not about to do something harmful to someone with an active support ticket moving forward. It will be deleting very small amounts of words. So it's not as bad this. This specific start is, you know, 2600 something orgs, that's a lot of orgs to start with, so we really want to make sure we review these manually.

A

I acknowledge that 664 orgs for you to manually go look at and I would normally script this, but I really want to be better safe than sorry on this one. So manual review is the best step um I'd like to ask all of you to be done by the end of day Thursday, so that I can kind of do what needs to be done for the deletion stuff.

C

Can we script like just to see which ones are like ready for deletion or whatever, and um no actually I guess yeah? We need to know who's got open, yeah, never mind.

A

You're, looking at one API call to get the org one to get their tickets list.

C

Yeah, because you can't just do you.

A

Know, oh they're requests right if they have more than 20 whatever 25 or whatever it is you're gonna have to make it for every ticket, and then you're gonna probably have to do another call to say. Okay now, I need the ticket here.

A

Dog filter needs to sit by me and start scratching and then you'll have to grab.

B

Ticket data to find the assignee.

A

And all that stuff and honestly, at that point, when you've made like three or four API calls for one or it was.

C

Probably quicker for you just to look at it.

A

Make a script and do something like that: I'm not going to discourage you from doing so. Yeah.

B

I'm, giving you 664.

A

Words to do so how you want to go about that I'm, leaving up to you to decide.

B

All right, uh I have a question Jason.

B

um It's about that. This list has some false positives as well like there is a Arc which is not expired right now. I can check that. um Definitely.

A

Marked like argument.

B

A

B

A

Because I'll review, all the ones that say active ticket do not delete. There should not be ones that are false positives unless they're not sinking properly in zendesk, which then would explain why they're false positives.

C

Even like solved, we would consider active right because it's not right.

A

If it's and if it's like it's been solved for six days and it'll close tomorrow, just go ahead and close the ticket. If you don't know how to do that, yeah, you have to use the API to do it, but um yeah, just kind of you know, look over and make judgment hold on that stuff and kind of figure it out. If you're, not sure.

B

A

Market active do not delete and I will review it um that one in particular you're looking at in a bill, looks like they reviewed and they renewed like literally today, which is probably why it's come across.

B

A

Yeah, so that, if you see those mark them active I will move them from that list there. A lot of that was I've.

B

A

This data very early this morning, so anybody who's renewed since then well yeah. They renewed since then like.

B

This particular person yeah.

B

B

Yeah but like there might be cases like that that I have marked them good to delete, but but by October 3rd they are like they have renewed it. What to do that? Yeah.

A

If they were new between now and then the script actually would not see them as eligible to be deleted and they would be deleted.

A

So, like I, this list came from what the script is finding as good as ready to be deleted.

A

So if they were new.

B

Then, when the script.

A

Runs it's going to see that they've renewed and it will not mark them as red HP deleted.

C

A

These are like already yeah I grabbed these early this morning. This is as of this morning if they were new between now and when the script actually does its first run cool. We don't have to worry about it and, uh while you're going through these, if I like, do not delete the org itself, because it's not the org itself that is deleted. We also to follow data retention, have to delete every ticket, every user and the org.

A

So unless you want to manually, go do all of that which I wouldn't recommend the script will handle it for us. uh Don't delete the work because I know uh that was my first thought is: oh I'll delete them as I go and I'm like no wait. No I have to do other things and I might as well. Let the script do that, but definitely Mark any false positives, so I can look into it to see. Why was that there?

A

What in that Salesforce account mayor, said that it's not active when it is, of course some of those could just be. They were new today, I wouldn't have caught them at seven this morning, but cool I know that's a pretty Monster task.

A

um I know, that's also not a fun task, but it is something we need to do so. I appreciate it. If you get stuck on any of them, just like I said, move them over to my tab. That's fine!

A

um Once I get in like kind of a flow I start knocking these out pretty quick, I'm sure an appeals, I thought, maybe I'll do a couple of them. They don't take that long to do actually, and most of them should not have active tickets. Most of them should be very expired and very much not active.

A

Cool uh any questions on all of that foreign.

A

I appreciate it like I said, if you get stuck feel free to hit me up, let me know um trying something slightly new, this time: I added the upcoming PTO, so obviously Alyssa's on print to leave until 2023 I'm not going to bother with exact dates, because 2023 is not that close until it is off on the 26th, which was his yesterday today, I don't know on time frame is Monday. Tuesday is Tuesday. That was this Tuesday I, don't know I'm just I time. Math is terrible.

C

It's so hard, it's like so hard and.

A

In New, Zealand just did like their daylight savings, so I was like. Oh, it's even worse, I mean he's a you know, he's Australia, Winstone um and I'll be off on the 18th. There will be more days in October I'm taking off, but that's the one coming up. The soonest and yeah.

A

Yeah say more about my days off when they actually get closer, but I just put that as an example, myself of yep I will be off on the 18th of October, which is in my mind, super far away. But it really isn't scary how close pocket poker is already to us.

C

Of course, yeah.

A

Any questions comments, concerns General feedback, any of that fun stuff. Anything you want to bring up.

C

Nope I think I'm good.

A

Maybe any questions comments, anything you want to bring up.

B

Nope I think that's completely. Fine I am completely okay with it. Sorry my net sucks I'm. So sorry.

A

No, it's all good.

A

I'll get dust tomorrow, Friday whatever day of the week. It is for you well.