GitLab Digital Experience, 22 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Digital Experience Retro - Sept 22, 2022

Description

Digital Experience handbook pg: https://about.gitlab.com/handbook/marketing/digital-experience/

A

Foreign welcome to digital experience. This is our iteration retro meeting, we'll go over the things that went well and some things to improve on for this past two week. Iteration today is Thursday the uh 22nd of September. uh We have things that went well and javi's got the first one.

B

Muted uh people are great, that's pretty much. The easiest way to sum that up, I just feel like people in this iteration reminded me, the the value of like including others, and just like being overall helpful and putting your heads together. um I felt like it's really great, that even after, like our new restructuring, like I, still feel like, we can put our heads together. So that's always nice to hear.

C

um Yeah just like plus one that for sure um and then also just a shout out to Carrie. Thank you so much for picking up um where Tina and Jess um just we're out this week and there was like a million and one things that um you're taking care of and following up on and just getting the engineers going. So no one was blocked. So thank you so much for that.

C

But um oh that's me again um and mark. Thank you so much for the team. Ups page, it looks so good. um I know that was like super tight timelines and Tina delivered. It really quickly, so shout out to her as well, but um it looks so fantastic and we're so ahead of schedule. So hopefully you get content soon and we get it live really soon. So thank you so much um and then Mateo.

D

Yeah um on mine, I want to say big things to Nathan for helping me with the figuring out what the product team wanted on the solutions page they were. There was a couple of people from product team contributing to the content that we needed, but they were mentioning features that when were not on our features, yaml and we could couldn't easily just add the features that they wanted. uh So yeah big thanks to Nathan for helping me with that.

A

huh To say uh it's not super work related, but it was really really cool going to the uh the Latin X tmrgu, which is renamed Dante I. Don't know how to say this, um but going to that and seeing Javi lead it. That was really cool. It's really awesome to see our own team uh contributing to the company and making it more inclusive, um very, very cool uh meeting that you you hosted.

B

Thank you. um We hosted something before uh that. Wasn't for Hispanic Heritage Month, maybe like a month and a half ago and I think we were like seven people were like okay cool like another one of those that should be easy and then logging in and seeing like 50 people I, don't remember how many exactly were in the call, so that was just like whoa. That was a lot of people, so uh that was great and thanks for calling it out appreciate it.

E

I think I have the next one I think yeah sorry I'm lost on the dock, everyone's adding stuff. uh We released the sync script for features and categories and I just want to give a huge shout out to Margaret and Tyler for helping with that. uh I know it's kind of behind the scenes, but it's going to have a huge impact and we're.

F

E

Finally, a single source of Truth for for hopefully uh features but yeah. Thank you. Miracle you're up next yeah.

G

I just wanted to thank Tyler Javi and you Nathan, because you guys helped me out with all the long story stuff and that would have gone so much worse. That I didn't have you guys help. So thank you.

A

We can move on to some of the things to improve on uh javi's got the first point. There.

B

Okay, this is like a there's, a lot of catch-alls for this, but I'll start off with just reading it like having a DEX team email. uh We could use a set of set up a hosts for the zoom meetings, use a shared login for various tools that we use, uh create decks open hours for people to collaborate and have zoom meetings. I see lots of benefits. If we just make that account and then beneath that I say, can we make an open meeting for people to discuss and collaborate together?

B

We use that a lot uh or we use that concept of like an open room. A lot like when we're running through the launch, directly stuff and I felt like we got a lot of benefit out of just having a place to like congregate, kind of like a like a plaza I like to think of it, as like, a plaza for people to meet and like collaborate. So I thought that was cool.

G

um I was just thinking if we did have like that next email uh Laura, you can finally give up your your uh meeting power rights to that email and then, whenever we need it, yeah foreign.

B

You want to vocalize your point in the yeah.

C

For the open, I don't like I know it's like we use zoom, but um there is like a slack huddle option so at the bottom left on slack. um Maybe water cooler is one because a lot of people are already in that. But you know that's not really like a work related one, um but it is hard getting contractors into any new slack channels that you create, um because that's just a bit of a process so um just for expediency. Maybe we could do like an in like a DEX incidents.

C

Channel and like just do huddle on there. I don't know, but.

B

Yeah yeah, we could try doing it in the water cooler and see if that goes well, um since we don't have to do anything we already have like everyone already in there. Is that something I'd be willing to try it um yeah Miracle. You got a sub Point C in that point, if you want to vocalize that.

G

People having access to things to actually finish out their issues, so if we could just make sure everyone has access to stuff before we assign them things, that would definitely speed things up.

A

And like it would be great for everyone to have seats or like accounts for everything, but in the meantime it would be awesome if we could like list somewhere in a Canon book who has access to what. Because I know like for a while. I was like I, don't know who has launch directly access, so I'm, just gonna think like three people and hope for the best yeah.

H

And if you don't have access, you can't even see the list of the access.

A

I

uh I got the next one um incident process. Should we Define something in our handbook or just keep winging it? um We had a couple. This iteration first was the launch Darkly um incident, but that got resolved um and the second was the unable to scroll main navigation, which was an incident issue.

I

um Javi I see you've got a point under the first one. Oh.

B

Yeah I mean just with the creation of incident uh channels like we can't add contractors that are assisting in that. So that's like another thing that was hard was like: Mattel was helping us, but we couldn't like have them in the channel. So that was just like an extra layer of like slowing us down.

I

Can we uh retire that channel? How does the team feel about that.

B

I feel like we could it just feels weird, because I still feel like we're using it. That makes sense for like launch Darkly related, like information I, don't know if we should just make another Channel. That's like a catch-all for that.

I

Should we be using our digital experience team channel, so it's public.

H

I think that's clear um I like knew that that channel existed but didn't realize it was like I. Never like I didn't like favorite it there's so many sex channels right so I, just like stopped paying attention to it and then someone's like. Oh there's, these conversations happening here and I didn't expect there to be um I. Think it's life. We should.

H

You know I I, think the best course event of action is to minimize slack channels as much as possible and like only add them, if need be and like I, don't know if, like I, don't know, if our incidents are so urgent that they require, like the slack Channel process that, like the product team has for their incidents right, like I, don't think they generally speaking out of a impact like customers at the same level.

H

I don't know if they have the same sort of like ramifications for the business, um and we also don't have the same sort of like pager Duty setup that that, um like the site, reliability, Engineers do right, like I. Think that going to like digital experience, slack Channel, where people expect to find us, and if someone is curious, they would expect to maybe see like a thread.

H

They're I think threads are probably as useful as we need them to be, and then like an Associated issue for the thread um because, like mostly I, think everyone's in that public Channel I think other people know if, like if they're coming to see what's going on with our team, because they know something is up, and it's on us I think they might start.

H

There I think finding the threat is useful and then, of course, having an issue is like the best way to keep it permanent, because the psych messages will get deleted after 90 days, anyways.

H

So if we do come up with a process, I'd like strong opinion, like new slack channels, shouldn't be a part of the process right, like threading issue creation and like like ways to elevate visibility like maybe we like post all, we l, we like make a thread in our Channel, share it right to like the marketing channel or to like a relevant one that, like a relevant channel, that is sort of an umbrella for us as well.

H

So like people who aren't who don't know that our team exists right but are like sort of like oh I. Think marketing right they're, like oh, that, like I, do see this right or even just what's happening at git lab. Something like that.

I

Yeah I really like like threads Bill's, uh you got a A couple points under A2.

C

um Yeah, that was my bad I created incident issue types I just thought that was like the bug ticket version in gitlab, um but yeah Laura and Lauren good points, the you can't add an iteration and you can't wait it um so I will not use those for future um like bugs that I'm trying to log foreign.

E

Just make sure that we're removing the same points that come in just so, we don't end up overloading and then I guess that there's a prioritization question go to philza.

G

um I think I have the next one. This is specifically about uh launch directly um just updating the handbook page. There is some discrepancies in the language that we're using like control versus experiment and control versus variant so swimming. If we wanted to Hash that out, while we're also doing all this other launched directly stuff.

C

Dennis, what do you use, or what do you prefer.

F

C

F

Just assume experiment meant it's like the name of the whole thing, but um in the case of like, if we're doing ABC testing, then maybe we do like variant a variant b, control right, I, don't know what you guys think.

C

Yeah I personally like variant but I'm, not.

F

Because the definition of variant means other right based on the pandemic, what I've learned about the word variant um should we do a poll I can just put up a poll and see if we can vote on it or if no one cares. I.

H

Think I mean you're. The person who, like it like funnels down to you right, I, think we should go with what what is useful for you.

F

Yeah, okay, we'll.

C

Go along with it great.

F

Are we cool with Varian any objection to that.

F

Variant just did a camera thing: okay,.

G

um And then I think Mateo has the next Point.

D

uh Yes, I was wondering I pick up, picked up this issue that I linked here um to create a new component and then maybe test it. uh But I was wondering how to test for this. One would work since there is no control component. The component is a new one to be added to the home page. So how are we measuring the success of this new component? If there is no control variant, the control would be the lack of component that Nathan just mentioned so yeah.

C

I can answer that. um So um what I'm thinking right now, when I need to run this by Dennis and Michael, but is I think we're going to test out the homepage full Carousel versus this featured content, um because the idea is then we can have three links on the homepage Carousel and we can have three links on this feature um content and we could have the same Three Links and then we can kind of see like which one is more effective and um directing users one way or another.

C

So that'll kind of help us gather some information, see which feature we would like to use and then in the future we can also try testing out like different types of content to be like. Is it better to use the homepage Carousel versus the featured content block? But the idea with all of these new features on the home page is that it just provides us and provides marketing a lot more tools to be able to use.

C

So we can keep the home for each fresh so like as new things are being delivered, we might want to use the top Banner or we might want to use the carousel or we might want to use a feature block. um So it's like we're going to use them in different instances and like mix and match them and there's a whole bunch of like testing that we could do down the uh down the line. But I think one of the initial ones we might want to do is the homepage Carousel versus the featured blocks. Does.

D

That help okay.

C

D

Yes, okay and I think I have the next one. This one is more of a personal opinion. uh I think having everything in the same place is easier to manage, uh at least for the a b testing issues, um maybe creating a board to just display a b testing issues with a tag instead of having a separate project. But uh yeah just would like to hear your opinion on that.

B

No need to vocalize my.

G

B

I just made a separate project for like what a board for that could look like if we use like a tag label if we use that being able to have like a view to be able to track all of that, um I feel like we should increase the clarity of what a b tests are running at the page, a single point in time, because I don't feel like people think that's very clear.

I

uh I, don't think we should have another board. I do agree with the label and then I will link to our handbook page where I've created a table where we can list what tests are running. What the winner was for past tests um and that's.

C

I have a question at what point um do we move issues like say we have the A B test and the A B test project. At what point do we move issues for like the creation of the work into the buyer, Experience Project versus like yeah I? Don't Lauren I think this might be in the process that you're like coming up with so TV.

I

I have so much to tell you. No I'll share um the Epic that I did for the recent one, um which is in oh gosh, I'll link it later, but we have an epic the a b tests go in there. You have your govern issue and that's the one that fills as a sign, and that is the issue that is describes the A B test, what we're doing and it's in the a b testing project.

I

You then have the create issue and that's where we're actually creating the code to run the A B test, release issue we're actually releasing it to production. You have the unreleased issue or you're turning it off and then Dennis has a manage I. Think it's manage issue where you're collecting the data, um so there's many moving parts to a Single, A B test. It's not just one issue and we can't all capture it in one issue.

I

um I will find that epic right now.

A

Oh sorry, I I didn't put my name down but I'm interjected. um As a quick question um like for the test, we just ran that um had the register button in the nav.

A

um We want to let that kind of um sit for 40 days before we like officially declare it the winner should that go in like the completed a b tests on the handbook now or should that happen after the 40-day soak 40-day cleanse.

A

Or we could put it in there now and then, if things change take it out, but.

G

um Yeah, okay I was sorry I'm interjecting to you. I was under the pre the impression that once the test is done it's when we update the handbook, um because we're no longer like collecting user data and all that stuff and then things change. Then we would update the handbook again.

I

I would like to update the handbook once the test. Is we don't have the launch Darkly code running.

H

A 40-day like cooldown period, though like if we don't have the launch directly code running how? How are we considering it, a part of like a an extended 40 day? I guess like what do we expect to be happening within like the 40-day period after we like remove the launch Darkly code.

F

um I can answer this, so we're pushing data during the collection period to Google analytics as well right. So after the test is done, we don't really need that data anymore, because it's already in Google analytics. We have to wait 40 days post data collection period because we're waiting for the user to complete their 30-day trial and then another 10 days to decide if they want to buy the product or not, and so that's the 40-day window. And then, after that we can look at okay of this data set from that.

F

You know four days that we ran the control. Had this amount of air or the variant had this many and then we compare the two metrics and see who wins that way. Gotcha.

H

Okay, so it is it if to remove the launch directly code like at the beginning of that 40-day period, and that 40-day period is like a like. A look back on the cohort that did get a B tested right and then we're looking back on that. And then you declare a winner like with additional data. Afterwards, I I was just worried that we were expecting to see launch Darkly information there.

H

But this is just like a cooldown before you like actually declare a winner, because you're waiting for the the long tail of possible conversions after free trials and stuff yeah.

F

Got it okay, cool all right? That makes sense. We don't need lunch directly after that data collection period, but we do need a way to flip it back if needed right. If the control ends up winning then revert back.

A

Okay, I think we're at item number four Lauren.

I

Maybe um yeah I guess just nesting issues up to our okrs and grouping them into buckets of work um that enables us to easily see like all the work that's been contributed to a fiscal Year's quarter um and I put the a link there to our current one. So um if it's contributing to our okrs link, it to you, know child or parent Epic,.

I

A

uh Yeah um I I, think some of my team anyway would have seen me adding filter to all of the customer facing Mrs that have come through for, like a final sign off just so she's able to communicate with all of our leaders for what's been changing and all the stakeholders, um but, like you know, we're just so efficient that we're like churning out Mrs and it's really not fair, to ask bills. That'll look at all of them.

A

um I think that's going to be like a bit of a bottleneck, um so I'm hoping we can have a bit of a better process for that. Since it's early days, Lauren just mentioned in my one-on-one half an hour ago, um a solution, a potential solution. So I can let her take that away, but it involves releasing stuff to prod but ignoring it in the the roads. Ignore um letting stakeholders do their thing and then we make any changes and then you know remove it from the ignore file. Something like that.

I

Yeah I will work on the proposal, so everyone can contribute, but the main the gist of it is we have. You know we are the devops lifecycle company. So let's use that process and our planning um and kind of add like extracting like we're, not just ux and Engineers. Everyone is part of all stages of it.

I

um So there's this whole planning stage where we're planning the page. What's going in how it's and they're working with stakeholders and that's a whole bucket of work and then there's the create stage where you're actually codifying it and you're putting it into the repo and then there's the whole release stage, which right now we're creating and releasing everything into one and it's getting a little muddy um and maybe separating that out.

I

So we create an MR and it's just one single commit turning the page on and then their stakeholders are reviewing the Mr and the coded template in a review app before it's released um and then monitor, which is data and Analysis.

I

It's all super rough, um but I'll try to wrap that up by this week and share it out with everyone in our team Channel and we can yay or nay it just an idea.

E

I think it's super cool, um I guess what I'm nervous about is like what exactly needs to be seen by um stakeholders before it goes out. Is it all new pages? Is it another release? Post was a sensitive one. I know, blog, probably is just seems to be like spotty, so I.

C

Don't know it is spotty and that's I think the challenge that I have too is like I, don't want to block the Engineering Process, because I do want releases out and like there's a section of the site where it's like, we own it, and we want to move quickly and we want to release stuff.

C

um But it's like the release post. um A couple of the events. Pages there's another one that happened earlier and I can't remember what it was um and so I know. Justin had a good idea of like when you do like AR approvals like access request approvals, they have like a label that says like stakehold or like your manager, needs approval or whatever um so I kind of like that of like in the initial issue.

C

Maybe we put like a stakeholder approval required and but the only thing like for me that I'm hesitant about that is like will that label transfer over through all the stages that Lauren just said and also like? That is the onus on the engineer to like check the issue to say like this is required before you release it. I, don't know um happy to try things out and iterate on it.

C

Ideally I'm not reviewing all the Mrs, because that's just a blocker and yeah, but I do also just don't want to release things with stakeholders and that need approval beforehand, because this is I think it's happened like two or three times now.

J

Is there a way in gitlab, either or bot, or something that if there's a label I know you, you can't merge if it says draft, but can you set up so you can't merge if there's a label specifically.

E

um I, don't think so, um but you could switch the draft status.

C

We could think about it like we don't need to sober this today and now, but yeah.

A

Miracle Do You Wanna bring.

G

Us home, oh I'm, so sorry um I'm gonna, bring it back to access just for the last Point um Megan suggested that we have like a page where we update, who has access to launch Darkly. um Should we do that for Google analytics too, because I can see that also being a bottleneck.

G

Cool that was, it.

A

Okay, I forgot to add one thing that went well I just stuck in it at the top I just want to say thank you to Miguel for handling all those B deep errors. um It was like eight Mrs, a bajillion changes, but it's still nice not seeing those warnings anymore. So thank you.

E

That's right does that future proof us from view3 like the new implementation. Is that, like view three approved kind of thing,.

F

I'm not really sure about that.

E

All right, no worries I'm, just curious I, don't know if we're ever gonna do it, but we'll hate to have to do it again.

F

Also, uh change was also only invite experience, so we still need to remove a couple of them on slippers. I will do the dish for that.

A

Cool okay, any last thoughts before I hit the stop.

C

Recording um there's just a few action items, but there's no people assigned so I, don't think they're gonna get acted on. So if there's something that you'd like to take forward, please add your name.

A

Okay, bye, thank you for listening.