GitLab Digital Experience, 7 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Digital Experience Retro Video - Oct 7, 2021

Description

Digital Experience Handbook Page: https://about.gitlab.com/handbook/marketing/inbound-marketing/digital-experience/
Digital Experience Retro Doc: https://docs.google.com/document/d/1kMNiUF2UDuSrMDuzLyRi8OEhVxry_MJoYi38RmmWafY/edit?usp=sharing

A

Hi everyone welcome to the digital experience team, uh iteration retro meeting we'll go over what went well and some things to improve on from the last uh two weeks um today is october, 7th and first up is uh jess.

A

I made my first and second commits this week with a huge thanks to lauren barker for serving them up on a silver platter yeah. It was so easy. If I had known it was that easy? I would have been doing them a long time ago, but more to come. Hopefully, uh nathan, europe.

B

Yeah, just a quick one, thanks lauren again for helping out with those interviews, it was just made it so much easier but yeah. That's all I wanted to say laura.

A

uh Yeah our little uh side outage today was you know, spooky and the first time since I've been here that we've had like a full kind of everyone. Messaging us like the site is down, um but within about 20 minutes we had a solution in the works, even if our pipelines take a little bit longer than we want. uh We found another workaround and it was just good to see everyone. You know doing things and trying to help. So the team.

A

Are we back to david.

B

No yeah back to me uh yeah awesome, tyler, that building to play stuff like I pushed abroad yesterday, and it took four minutes and eight seconds like I literally got up, came back and it was on prod, which was unheard of. So I'm really excited, and now it's nice to know that if something does go wrong like we can get to it almost instantly.

A

Yeah, how long was it taken before.

B

uh I think between 10 and 20 is what I saw.

C

Yeah it's hard to gauge because.

A

C

Depends on a lot of other factors like independently, like 10 and 20, but like you, could uh very possibly get stuck on long train and take 40 to 60 minutes. um So that's always a risk here. You won't uh until you do because, like eventually like you know, this is just a fundamental like problem of like you know, entropy in the universe like as this grows in size, the time must also grow and as it grows in complexity, the pipeline must also grow and comply like we will.

C

You know if we maintain a similarly sized and complex website, then this will eventually turn into that too. But it'll be nice for, like six months.

B

Yeah, let's enjoy it, I'm making changes every hour just to say it. I did laura back to you again.

A

Because you kept stealing my lines in the dark, you kept it yeah.

B

I know it was funny very rude, even lauren. I think I was bugging her earlier too.

A

Get out of here we're gonna, lock nathan, out of the docks from here on out um yeah uh investment vane. I made my first, mr. I haven't mercy yet so I haven't seen the blazing fast feats, but even just um the organization of the the core marking site everything's easy to find it's lovely to work with it's cool and I enjoyed it so good job to everyone involved and then parker.

D

uh 10-Year anniversary looks great, thank you so much tyler for getting that done. It's awesome to see that beautiful event template get put to use. It's really really good.

C

Thank you thanks for everyone for doing the event template it has been uh the the building of it has been. The easiest part like the thing that could have been the hardest has been smooth. uh So that's great. I appreciate everyone like I said it in the release. Video before like laura, like your countdown component, like they're like hey, can we get a countdown? I was like sure, can whatever here's the countdown I was like laura did it. I don't do this it's great and the same thing for like every other component.

C

In this thing, like it's been, it's been awesome, so uh yeah uh big ups to the whole team.

A

Cool uh things to improve on, I think, even as the first.

B

um Yeah there's been a lot of changes to nav recently, I think we changed some sizes and stuff, let's just make sure we're applying it to both um both repositories and then, as like a a bigger thing. As we build out, I guess core marketing site. uh We have to make sure. I know we talked about buttons. Yesterday it was a topic of discussion, um we're gonna, try and make those the same in the old and slippers and core marketing site.

B

So just a heads up going forward, I'm going to try and like maybe once every sprint go and like find some element and then try and make it cohesive between everything. So it's going to be a bit of manual work, but I think it'll be worth it in the end, so yeah, let's just try and make sure that we're uniform.

D

I have the next um and uh it's just to default, to building new stuff in the new repo. So if we got a new template just default to building it there- and that's like a personal thing for me too, like breaking the habit of like oh.

A

D

Doing it over here for a long time, I'm gonna do it this way, oh gotta. Do it a new way, just like always be thinking like that? uh Laura.

A

uh Yeah, uh you know with the on the subject of failing pipelines, um I've been seeing weird stuff with my mrs. I know. Javi also showed me one yesterday that had a couple of random failures and it just re-triggers itself and, like you know, wears itself out and then eventually it passes and everything's fine. um It just seems strange: okay, yeah I'll. Let tyler jump in.

C

Oh yeah, just um you know, I don't think we know the the root cause of this, but if there's a pattern of a lot of apis, a lot of it is like get lab rate. Limiting us, gitlab and like chad has like started an issue to investigate it. um The nice thing is the more that you do in the buyer.

C

Experience repository the less you'll run into this, and so that's kind of the hope, and what I would also hope is that there's a second order effect of so like you yourself can get around it by doing work and buyer experience, because buyer experience is smaller and like less frequently contributed to so it's running these things less often so you're unlikely to get rate limited in the same way, you're, probably not even hitting the same api endpoints because we're not there.

C

um So you can just like sail through that's because like right now, it's like okay. I want to make a one line: change, okay, that one line change has to hit all the apis that the pipeline runs, don't need it! um Buyer experience.

C

Does none of that and it'll push directly to it and then the second order effect is the more of those little one-liner changes and, of course, you'll have to like first make a 200 liner to like rebuild the page, but like the more of those that run the fewer of our uh the fewer mrs are running in the www.getlab.com repository, the fewer api calls are being requested.

C

We reduce the load on the systems that are rate, limiting us right um and yeah uh javi next partial builds but not and he's asking in the chat, but not in this way. um We probably we might be able to figure it out, but like right now, no, fortunately we're small enough right now that it won't impact it.

C

But I think we can actually reduce the load on the services that are rate limiting us and hitting these problems by like moving our smaller changes into something that doesn't have to hit these apis right and like eventually, the the www repo will shrink down to a place where the only thing that's running on it is the stuff that needs the apis, which would be awesome.

C

And then those will have a lot more overhead is um that's just off the cuff, but like uh yeah we're seeing a lot of that stuff, and I think it just has to do with rate limits and increased load on that. So.

A

Do we want to do action items does? Do we want to talk about hobbies thing in the chat um time based pipeline setup so that fixes defaults being ran every so often, I think that's about the random changes. Maybe button changes that we were talking about.

C

Yeah, I don't know what this is bobby.

A

Are you able to talk.

C

C

Docs runs every four hours: okay,.

A

Yeah, it might be something changes.

C

Oh, I actually mean you lump them all together on a run.

D

Yeah so yeah you deploy once you know a couple times a day right constantly.

C

Right, which I think was explored for this original, like like a year or two ago when the I've looked through the issues um uh when, like we were like really trying to get down from like 60 minute pipelines like that was a proposal right. But then, like you, you smooth out the the time, but you like give everyone a minimum amount of time that, like that they're changed like they'll, never get a 10-minute deploy, they'll always get a four-hour deploy. So like it's more consistent.

C

But then everyone gets a four hour deploy and like you could do exceptions to that rule but, like then everyone's an exception, and then you get the variable, deploys between ten minutes and four hours right. So um I think, like with the volume of change I I don't think we should do that, because I think that, like it won't solve the solve the problems.

A

Cool, uh do you want to go over the action items or just.

C

Yeah I'll vocalize real fast that um we should do blameless root, cause analysis for what happened today with our outage with npm's outage. um I like made a comment in the in the incident issue. um People can hop into. uh I think that we got struck by lightning.

C

I think that this was like a perfect storm of things, and I don't think that we're that we have a ton of risk there, um but I think it's basically like five things like five step process, which is mr happened past its pipeline goes to master pipeline between the time that that happens and master pipeline starts.

C

Npm goes down, master pipeline is all running already running and is unstoppable running independent parallel jobs, some of those parallel jobs build like pointing to assets assuming that they're going to have assets don't get built because npm is down incremental, deploys, go out to the to the bucket and like reference, these non-existent style sheets, and then everything breaks so like that wouldn't have happened if npm had gone down and failed on that. Mr, like 30 seconds sooner, because that mr would have failed and not made it to master so like.

C

I think that that's just a race condition is my is my thinking on this. I don't have any evidence to prove it other than like looking at the cicd panel and sort of like comparing some stuff.

C

So that's that's my thought and I think we should do like a bigger analysis so that we can either um so that we can like get down to the bottom of that and figure out if it makes sense to to do anything because we we could build out some more resiliency and we could build out um snapshots of our deployments and like save rolling versions of the website across google, buckets and swap over as needed, but like that would be a lot of effort and like we already have a ton of like we're already extremely resilient.

C

If an mr fails, it never gets to master and we're always just defaulting to the last successful master. Build like this is always the case. Whatever got out is like there. So I think we have a ton of resiliency and I think that this was being struck by lightning, and I don't know if it would be worth the effort to to do the other stuff, but at least not right now, right.

A

One question that I think came up in the incident channel from today from michael frederick um said: you know: do we have monitoring um and alerting for about uh for like assets or css, css and other assets? um Are we able to set up? I don't I don't know what the procedure is. I assume that throughout the website, there's a ton of 404s for random images that are missing and whatever is it worth like figuring out, which ones are legit yeah.

C

I don't know I mean what like we do, so we do have monitoring because we get it through log rocket. We get it through. Google analytics like we know where we get 404s and we get reports like hanef often hits us up about like hey. We have this thing missing and I think, like it's awesome like you know, people are on top of it. We don't have alerts the way of like outage alerts, because I think exactly what you're saying is like like one: where do we run it?

C

It's all client-side like where the 404s happen like do we have like a server that pulls the site? You know every so often and like goes through every single page and then lets us know when it hits a 404 error in a browser that is running and then what? If that's like a local failure and then do we get 100 errors that one computer couldn't get it stuff, because, like of a network error on its end and everything else is totally fine right, I don't know.

A

Yeah fair sounds like more false positives, false negatives, false positives than what we need, but yeah cool, okay. Well, I think anyone has any final thoughts, um yeah, otherwise,.

D

I got I got a little tidbit I was surfing. Yesterday morning I saw a great white fully breach jump, get a fish nice paddled in calmly yep. That fish could have been. You.

A

D

Could have been dinner, my 35 years of ocean experience, I've never seen a great white. That was the first time. That's cool, that's scary, very cool. They look very big, very efficient.

A

All right I'll start the recording there with that perfect little send off we're we're lucky to have lauren here today and we'll see you next time.