GitLab Support Operations, 18 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Support Ops - GitLab.com Projects - 1-1 Issue Generator

Description

Jason Colyer, Support Operations Manager, takes us through how the 1-1 Issue Generator works

A

Greetings: auntie callum support operations manager here at get lab and today we're going to go over how the one-on-one issue generator works. So let me share my screen and we'll get started, so this is the repo where the one-on-one issue generator lives.

A

Now the way this works is through using gitlab ci cd to run various jobs, and these run scripts within it. The stages for this are test gather report and create.

A

There is the cache path of vendor ruby to help kind of speed up some of the gem installation. um Every script will run before it actually runs the script that it said to run. It will run ruby, dash feed from the version and then bundle install to install all the gems.

A

Now the test stage consists of one job: it's the test files job. It doesn't run on schedules, but this runs the test script every time. This is useful because it makes various tests we'll go over that a bit, but it prevents us from putting bad files into the repo. This helps ensure that everything runs smoothly when it comes to the actual ci cd schedules.

A

There's the gather stage. Now this runs the gather data job. This only works on schedules, it requires the variable stage equals gather- or rather this variable stage has to equal gather for this to actually proceed.

A

The script it runs is the gather script we'll go over that in a bit. This creates an artifact in the data folder and it expires in four hours. Now the report stage runs the daily report script. It requires the gathered data job to have succeeded. It only runs on schedules. It requires that the stage variable is equal together and it runs the daily report script.

A

Then we get to the three create jobs. These run based on the variables that are set. There's the weekly reports. What this does is run via schedules when the stage is equal to weekly and it runs the weekly reports, script, the monthly report, job kind of the same deal. It requires that the stage is equal to monthly and the agent name variable has been set. It doesn't check for what it's set for, just that it exists. It has value, it runs the monthly report script and then there's the quarterly report script.

A

It's pretty similar to the monthly report script. The stage has to be quarterly in this one and it has to have the agent game variable and it runs the quarterly report script now going over these scripts. Let's uh look through these real, quick, we'll start with test, because that's the one that has more data to it, but essentially what this does is. It requires the generator files in the lib folder and it's going to go through and use ruby to search every ammo file and ensure that it is readable. It can be parsed.

A

It's going to look for all the announcement files and once again just make sure that they're parsed make sure they have data, make sure that they have the announcements array, the comments files it's going to go through those and it's going to make sure a they're parsable. It's going to make sure that they have the ticket review boolean it's going to make sure they have the comments array and then it's going to go through all the erp template files and it's just going to confirm, hey. I can render this and it works all right.

A

If there's no errors it'll succeed, but if there is any errors at any point, it's going to abort the function. It's going to fail the cicd job, which is what we want. We don't want to proceed with anything else if there's broken files in place now beyond that, you have the uh we'll go with the uh daily or yeah. Let's go with the gather script to start with, because it just runs. Generator client gather we'll go over the actual lib folders a bit, but the rest of these kind of do the same thing like.

A

If we go to the daily report, all it's doing is generator get lab. Daily report commit daily reports. If we go to like the quarterly report, all it's doing is running generator. Get lab quarterly report run monthly reports, gonna run, you know, get generator, get lab monthly report run, and you know we have the weekly report, which is gonna, run generator, get lab weekly issue create so going into these. um We start with the generator rb file. This is kind of the main file defines the module.

A

All it's doing is setting the module generator requiring the bundler setup requiring open, uri. Making sure bundle requires everything that it requires and then it's required it's uh requiring the generator client script.

A

So if we go into the generator folder, we can look at the client script um here. It's going to call the module generator and create a class called client. It's going to require the get lab page and zendesk files.

A

Beyond that, there's the gather function, which is the important one. That's where it gets the data it puts it to the user's value and then it's going to also use the you know. The zendesk global runs and seo's federal, run, pager duty scheduled and the get lab gather run functions to generate all this data and then it's going to artifact it. Now um I put some retry options in here. This basically makes it so that if it hits a timeout or hits a you need to wait kind of error.

A

It actually tells faraday go ahead and do that, let's retry, but you can look into faraday options to kind of learn more about this. The artifact function just takes the data. It's going to set a variable called date where it's basically just getting the date today in utc minus one day. So it's basically saying give me yesterday's date um from there. It's going to open a file in the data folder with the date underscore analysis.json and all that's going to have in it is the data that we've gathered, but in json format.

A

Now we said some clients, like the zd client, usbed client. All this is doing is calling the zendesk api client passing some values to the configuration, the pager duty, client, doing kind of the same thing, but this time we're using pure faraday for this we're making a new faraday connection to the api.pagerview.com url, making sure the retry is set with those retry options, making sure authorization is the token that we need that the headers are told to accept uh the vnd.page duty plus json.

A

That's something pagerduty themselves says to do and we're making sure the adapter in this case is net http. Normally I would go with the default adapter, but pagerduty was pretty specific that that's what you need to use for this.

A

um We also set up a gitlab client once again, it's a new faraday connection, using a specific url with the retry options. The url is coded. It tells the adapters the faraday default adapter and the headers. It sets the private token to the get lab token.

A

Beyond that, we have a function for the support team animal. This is basically going to take the url for the raw support gml file. Send it send that request to get through the gitlab client using the public, send get function, and it's going to give it back and save this as yaml.safeload, which means it's going to safeload it I passed in the date.

A

So it knows that some of these values in that file are dates not just strings daily reports kind of the same deal it's passing to the daily report repo, but it's going to use a specific date to figure out what file to grab it's going to make the gitlab for you know, get lab client, it's going to do a public send and it's going to parse that file in json, because the artifacts that this all generates that we might have to gather later are json files.

A

Now, there's the agent agenda file function. What this does is basically look at the agent and grab their uh project id for the one-on-one generator.

A

It's going to set that url there and look for the repository files in that project and if they, if there's nothing, it's going to return saying yeah. There is none false um if it does get something. It's going to json parse that if uh there, if it has a message to the json body, which would mean that, um but instead of the actual file we were looking for. What we got was a json object from gitlab. That said, no such file or whatever it's uh it's going to instead return false.

A

But if all that works it is a valid file. It has content. It's going to give us the content of the file.

A

We have the users function, which just says prepare users, and all this does is use the support team, the ammo file, the support team, yaml content. We got before and set up some basic objects for the various data we're going to gather. It varies based on what we're doing like zendesk has assigned public comment. Private common s sat and see cc's.

A

The default object is current and previous. That's something we use for the git lab because you can see in git gitlab we have parents default internal default, et cetera, et cetera, et cetera, and the pagerduty's object is just this week next week.

A

That's all the client file of the client class actually does, but uh let's go through some of these other files, I'll pull the git lab I'll, pull the gitlab one first. So we'll go through that all this does is it's making a new module called git lab and requiring all the get lab files, and if we go into the gitlab folder, we've got a bunch of files here. So, let's start with gather, we call the module generator.

A

We call the module get lab and then we're making a new class called gather for the client instead of just calling out the long string of generator client geoclient every time we're just going to go ahead and set the global client variable to mean that if you're not familiar this right here where it's two pipes, an equal sign is a ruby, a ruby kind of shortcut for saying. If client has no value, it hasn't been set set it to this.

A

If it does have value, don't worry about it, so we have the run function, which essentially is going to output that it's gathering from git lab. It's going to use the data from the agents, and it's going to loop through on every agent say that it's gathering for that agent's name. It's going to look through all these various objects.

A

Now I'll go over. Like one of them, because they all pretty much are the same, just different urls, um we have some functions just to give us some. You know the week start the weekends last week's start. Last week and last month last month, yeah they're, just a bunch of date, funk date calls that way. It's easier to call this. um We have ones for quarters which are a little more specific, but next we define the request, which is going to use that client that gitlab client to make a public send function.

A

It's going to use http method, the endpoint in the params and then use oj to load it for us. So it's nice and easy to work with agents are set to whatever the generator client users. Are it's just an easier way to call it?

A

We set some functions here to say this is what an issue object looks like this is what a merge request. Object looks like for a determined wait function. It's basically going to look at the merge request and look at the weight and see or look at the labels and see if it can find the weight label if it doesn't find one. It's just going to say zero. It has no weight.

A

If it does, it's going to actually split to give us what the actual weight is now um here we have some options that we're going to pass in, because, if you remember correctly in the request, we can you know, method, endpoint params. We are actually going to use some of this in the url, so the endpoints, but yeah like for closed issues. We want the status close, the assignees, whoever we're working with per page, give us a hundred order by updated at sort by you know descending merge requests. We want scope.

A

All state merge for page 100, author id is whoever we're working with and open options are going to be scope. All state opened per page 100 author id is whoever we're working with so um I'll go through pairings, but the rest of the calls, the internal, the merged the documentation they're all going to be the same. Just with different request urls, but it's going to print out what it's doing so gathering pairings it's going to set its options to the closed issue, opposites, pairings or closed issues.

A

It's going to get the results from a request, a get request to the project that is holding the pairings, and it's going to be looking for issues with the ops it's going to loop through each of those. If the close that is nil, it's going to assume if there's something wrong with it, it hasn't been closed properly and it's going to skip it.

A

The closed at it's going to use a date parse to parse the closed at that's because what we get from gitlab is a string, but we need to make sure that we have it in date, format for comparison with our other date objects if the closed at is before last week's start. Since pairings are for we report on those weekly, it's just going to skip the iteration, because we don't need a report on it.

A

It's going to print out a dot for each one that it actually reviews at this point just so, we can quickly look and see where it's at, um but basically from there if the closed at is greater than or equal to. Last week's start and it's before last week's end, it's gonna, say: okay, cool put it in the previous, reeks you know, objects the hat, the array for that.

A

If it's uh you know after the last, uh this week's start, but before this week's end go ahead and put it in the current um you can see like with internal it's the exact same general, gist of things there's some little nuances to it, such as this one only looks for success or failed because of internal issues.

A

Interviews are pretty much. The exact same modules are the exact same. Open issues are even simpler because it's just looking for open issues. Most of these are pretty much the exact same. We haven't analyzed, merge functions so that it can kind of go through. Merge requests to figure out okay, is it documentation front end back in design, support fix or other, but that's really all there is to the actual gathering of the data for git lab itself, so we can go back to git lab and let's look at daily report. This is a simpler one.

A

It's just making a daily report class once again we're calling the client as the gitlab client we're setting that request function again and then we're basically just gonna. Take that uh daily report and we're going to commit it. That's all there really is to it. It's essentially committing it to a project that way we have the artifacts. We can save them. We can call them readily.

A

This is actually an optimization we made since b1 and v1 just gathered them daily and reported on it that way or tried to gather them over the course of a week. um We found that often caused a lot of problems, especially since uh the search we were using for zendesk had a limit to the number of returns.

A

Wasn't great. um This was actually a fix where it's like right. If we just commit each day's thing, it's a lot easier to run because we can, if we need to break searches down by the hour by the 30-minute mark by the five-minute mark before we were really limited. So this was a good fix uh beyond that, there's like the weekly issue.

A

The way this works is it sets the class weekly issue once again we're saying that gitlab client the request function. Good optimization was, I should probably have another file that has these common things in there, such as the client, the request and have it have these files inherit from that. So that's a future optimization. We can do data it's going to do this by getting the analyze reports function, output for the title. This is literally just a function to say what the issue's title will be: a weak reports.

A

It's a function that is setting this week of reports to fetch reports which is literally going to take today and then go. You know one week ago to today and it's gonna map that to a uh a string and basically get every one of those report, files that the gather script or the commit script had made, and it's going to read those and it's going to push those to the reports array, which then that array is what it will output so or sorry.

A

What will return so week of reports is basically just literally a week's worth of reports of the analysis. Files of the you know results files, um one important thing: we do. You can see how we have analyze reports. So analyze reports takes that week of reports that we got up here. It takes the very last one because that's going to be the one with the most accurate support team of data um and what it's going to basically do is say well.

A

Okay, agent zendesk is actually equal to combine the reports because we need to combine all these various zendesk datas, because I mean we have seven days worth of zendesk data that we need to combine, and then it's basically just going to give us the last report now the way this works. When you combine them, is um it's going to make it a a loop on the main in the us federal instance, because it's two different zendesk objects from there? It's going to make an array on sorry make a loop on assign public comments.

A

Private comments set ccs, the various categories for zendesk objects, and it's essentially going to set the instance it's going to set that type and it's going to empty out the array. This is pretty useful. Just for the sake of what we're doing is emptying it out so that we can then go through every report. Add to it, but also do a bunch of checks with it. So um we're gonna set it in time and a start time. Then we're gonna go over the weaker reports. We're gonna set it to an array, go over it.

A

um If it can't determine who owns the actual item in the we in the week of reports, it's actually gonna skip it because it's like I don't know who this is no point continuing from there it'll go through that. You know that week of reports items that reports, items for the main and us federal zendesk instance go over each of these, and it's essentially just going to parse them to make sure that the updated date is greater than or equal to the start time.

A

The updated is uh less than or equal to the end time, basically making sure it's in that criteria of time that we want and it's going to push that to tickets and at the end, it's going to concat all those together to give us one zendesk object that contains a week worth of zendesk objects.

A

um So beyond that you can kind of beyond that, we uh we kind of get to where we're actually going to do something with all this data, because right now, we've mostly just parsed it gone through and all that fun stuff.

A

So uh the user template this is a nice function, basically to tell us what template to use for the user, namely it looks and says: okay well, if they have their own customized file, use that, but if not we're gonna go with the defaults weeklyreport.erb file run today is a function. That's just going to figure out what today's weekday is. Number weekday number is zero being sunday six being saturday, and it's going to look at the agents. One-On-One generator settings for day of the week and say yeah only run if they want it run today.

A

um This way in this way, we can make it. So if you want your report run on sunday cool, we can have your issue generated on sunday. If you want it on, you know thursday. We can generate on thursday and basically lets you kind of customize it. So it'll only return true, if that, if today, today's weekday is the day that you set it to otherwise, it's going to say false now the create function.

A

What this is going to do is go through the data for each agent and um if the agent's one-on-one generator section, is completely empty, it's just going to skip the iteration because they obviously don't want one made. If the agent's 101 generator's project id is empty, it's also going to skip the iteration, because if we don't have a project, we can't make an issue and unless run today, returns true, which means they want it run today. It's also going to skip the iteration.

A

Now beyond that, it's going to output that it's creating the issue for the agent it's going to make an erb renderer object, which is going to be erb dot new, and it's going to read the user's template file. We have the issue ops here. Where we have the title. Is you know the title function? The description, which is the big body of the issue, is going to be the er erb's renderer, the result with the hash users.

A

The agent data users is the data that we have entirely we're going to make sure to set it confidential. To true we're going to make sure to add any labels that the you know, the agents set in their one-on-one generator section and we're going to make sure that the agent is assigned to their issue, because we want to make sure that they know it's their issue.

A

So we're going to run the create issue function using the agent in the issue. Ops that we set up here to get in this should return an iid, an issue id and from there we'll run the comment on issues using agent and the id so comment on issues. Pretty much is going to set an empty con, an empty array as the comments variable and then it's going to basically go through the comments file, announcement file agenda file and then it's going to go through all of these and create comments on the issue for it.

A

So the comment file, um basically it's going to say the ticket review. It's going to have the ticket review function. It's going to have the from comment file function. It's going to concat these to return, those to sorry return. Those to the comment on issues function and we set a slug function here, just because it's easier to say: yeah, the agent's email, but the first part before the at sign it.

A

It gets tedious to write it over and over and over again, so the actual ticket review function that we saw above is basically going to say. Well, if the agent has a comments file, basically, what we're going to do is we're going to make sure we can read it and we're going to return an empty array unless they've sent take review to true now.

A

If it can't read the file, it's going to tell us, I can't use the file I'm using the default file and that'll just grab random tickets for the agent and the way that works is it's going to grab take all the public comments from the mainz in desk instance run unique on them map them, and it's going to basically map them to this string right here.

A

It's going to do the same for the usfed instance, it's going to concat these into one giant array and then run sample three, which will give us three random from this list.

A

Now the from comment file is going to look for the comments file. It's kind of the same deal of. Can I load it return like any comments they have, which might be an empty array, could be comments. If it can't read the file, it says, I'm going to use default, it'll go to default. The default comments, file and it'll grab put the return, those announcement files the exact same deal. Basically, can I read the announcement file?

A

If I can do they? You know return their comments. If they can't, then we're just going to have an empty thing, because we don't have any announcements by default.

A

The agenda file. We saw that in the clients function where it basically is going to grab the agenda file and give us the content if it has comments, cool return them, but otherwise it's just going to. Basically, you know, give us an empty array.

A

You see the empty file function here. All that literally, is doing is once again calling to it making sure that it's not empty and then the empty file, ops we have, or so I'm sorry, I'm misreading my stuff, my own stuff, the empty file function is actually going to make a post request to their repo to empty out their agenda file, and that's because it's supposed to empty after the report is made and the emp file ops, we have the branches master, the commit messages reset agenda and the actions are it's going to update it.

A

It's going to update the agenda.md and the content, it's updating with is blank.

A

Beyond that. We have the create issue function. You saw that called earlier. All that's doing is creating an issue not very original and aiming, and we have the create comment which is taking the id and creating a comment on the issue we've already created.

A

That's all. There really is to the actual weekly issue, we'll go over the monthly issue. Real, quick, but you're, going to see it's very similar. We've got the request. Data run is going to check if it's a valid agent. You call the data function, generate an issue.

A

Analyze reports in this case we're having to combine zendesk and get lab, whereas before we just had to combine zendesk but uh you'll, find it's not very different.

A

um The fetch reports, instead of it being seven days ago, right we went a month ago because this is a monthly report and the quarterly report will basically say a quarter ago, but um I don't I don't know if that's an actual ruby function, so we've defined what a quarter ago means um here we set the title check if agent is valid, it's basically going to take the environment that agent name that was required to run this and confirm that it can find them in the file in the users.

A

If it can't it's going to say yeah, I don't have a user, it's going to raise an error in the function, otherwise, it'll return. True, the combining zendesk reports. It's going to look almost identical to what we just saw combining git lab reports is pretty much the same thing. It's just squashing all these together and you know: that's that's really all there is it's concurring them together. Essentially, and then we generate the issue and yeah. That's all there really is to this. The quarterly one is going to be the exact same kind of deal.

A

So uh beyond that, beyond the get lab files, there's the page of duty files um once again like we're setting the client to the page due to client requests we're setting that um the current date start date. The current end date next start date. Next end date. These are all just date functions to give us the date range.

A

um The current options are the next option, so the current on call the next on call we're gonna go through these schedules using the static data, on-call information and uh yeah we're just going to determine who's scheduled, so this is going to go through them and determine who's actually scheduled for what date and add that to their pagerduty object and then return this back. That's all this is really doing so. It's a relatively simple file, the other one to look at would be zendesk, which is just calling to zendesk itself.

A

It's calling these zendesk files within here we'll go over global, because us federal is very, very, very similar, with the exception of the url. Basically, so uh we set the url to the get lab. You know the getlab.zendesk.com agent. We set the title to gatherings and desk global tickets. We just output. That client is the zendesk client. The data is parsing. The tickets we'll go over that in a bit, the artifact file said desk global.

A

Only it outputs that to data to json it's good to have this artifact handy agents, we're calling from the users and agent's zendesk object is going to be agent, zendesk mate. It's it's quicker to just make a function for this than to actually write it out. Every time.

A

Beyond that, we've got the start time and the end time start time is going to be the previous day utc at midnight. Basically today and then today's midnight is the end time uh from there we're going to parks tickets.

A

So what we're going to do is do a zendesk api collection and what it is is we want to pass the client I'm going to say we want a zendesk api ticket object, that's the objects, we're working with we're going to sort by the updated date, sort it by descending, and we want to include organizations and users. This is so. We have all the information we need for.

A

The tickets and from there skip the iteration if the updated ad is, after the end time skip, but if it's before the start time print a dot to show that we're reading it and then run the analyze ticket function, when it's done, it'll say done and it'll return. The entire agent's object.

A

Now: analyze tickets, it's essentially going to handle the assignee, cc's sat and parsing. The comments handle sign is pretty simple. If the assignee is blank skip the ticket, if uh otherwise, we want to use the agents that we have to determine who the assignee is. If we can't figure that out, skip the ticket, otherwise add it to their assignee object. Assignee. You know hash array, I should say um ss assignee is kind of the same deal. It's basically.

A

If you know we can determine the assignee put it in their s, app, because it has sad and now that'll come down when we get to the handle. Sf function handle cc's. uh Basically, it's going to look for the follower ids on the ticket and find the agent itself and add it to their cc's. That's all it's going to do.

A

If you can't figure out who an agent is it's going to skip them, handle sat it's going to return if the asset score is unoffered or if it is offered, because we want it to be good or bad, we need actual asset, not just we've not offered we have offered and that's where we get to the sm assignee function that was up here.

A

Parse comments is going to be the pretty much the exact same. It's going to go through the comments, it's going to include the users and, if it's, if it was created after the end time or before the start time, it's going to skip it otherwise, it'll analyze. The comment analyze comment is going to figure out the author and that's going to use it by detecting from the object. If it's empty we're not going to proceed.

A

If the comments public push it to that person's public comments, if it's private put it to the private comments we use, if else here, because if it's not public, it is private.

A

um We have a ticket object here, because you'll see a lot of you say: push ticket object. The reason for that is, we want all of this information here and then for organization, it's a ticket organization id or organization.name, but that became a lot to type since some tickets don't have organizations beyond that. We can look at the us federal, real, quick, but you'll see this one. um I did a little smarter because I told it to inherit from global, so we didn't have to define as much so.

A

In this case, I need to change the url, the agent zendesk, the title and the artifact and the object, but that's it. We don't actually have to change anything else, because everything else was defined in the other one.

A

Now uh real, quick I'll go over this real quick, we have the schedules uh generate one-on-one issues daily. This is where it's going to check to see. If it should make one-on-one issues, it gathers the metrics daily. It creates it three. It gathers at one thanks to runtimes. We know that that's plenty of gap, it's actually probably more than enough.

A

So that's all there really is to the one-on-one issue. Generator um by all means feel free to go through a lot of the other stuff like the gym file and the dot ruby cop to see kind of the things I said to ignore, because I don't want to actually check for those. When I do my own syntax checking um feel free to read the repo file kind of learn through it.

A

You can even make your own. You know you can even put your own one-on-one issue generator stuff in the support team yaml files, so you can have them generated for yourself and kind of see what you're looking at um up to you, but by all means, if you need any assistance, feel free to reach out to your fellow support. Ops, team members or myself are always happy to help. I hope you found this educational and I look forward to seeing you in the next video.