GitLab Memory Team, 20 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 02 20 Memory Team Planning Josh and Craig

Description

Internal doc link https://docs.google.com/document/d/10Q4gSHTvfR-Z6LgnvwBjnCn_-JNDNRPLth3wMr2X7rU/edit#

A

B

So we talked about in the memory team channels, so we've got the puma epic, nine five four. Now that boom is out on production. There are some.

A

Yeah, it is good, that's a great accomplishment. No, it's huge. It's a long road to get here. What we did it yep.

B

So this is epic 954, so enabling boom on the web server, which it's kind of turned into just the Puma epic. That's not just focused on web server get loud. So Camilla raises a question about what's next, what do we need to wrap up for Puma and I? Think these five yeah top five issues? It makes sense for the most immediate need.

B

The rest of them are nice-to-haves and we just need to make sure that they're in the right priority order- and we have folks assigned to that like the first one- make come on by default- is not scheduled.

A

We can't we can't do that. First, that's gonna, be I, think a larger thing right, so I think we first need to figure out what the right defaults are for omnibus, like number three. That way that way, we can have a like-for-like installation that makes sense and then I think we're gonna need document up to the documentation to make sure that we have I. Think I. Think I opened up an issue on this, like maybe I'm fighting at it to the document to this epic, but we need a so that documentation.

A

For you know, people are different.

A

No, no sizes right like what should they structure that note that worker at thread counts? Is it always like WX and like t2, or what are we? What do we? What like? What do we land on? Because that's that should be in that zone, because we have customers with some larger nodes, of course, and we want to have some documentation in there, a cycling restore defaults which will likely be like the you know, it's pretty small box installation right.

A

B

Me try and find that who.

A

Mona documentation, yeah document worker thread site for Puma. Here it is so it's pretty brief. It's a Seinfeld at nine. What's the app what's come up or get that right? Okay,.

B

I got in 54.

A

B

A

Polly right now: okay, cool that is added, awesome; okay, so that I can meet down below.

A

That one nice we have we have, we have two on, but okay want. One is the charts. Let me rename this to be chart. So it's more clear.

A

The ARBs commented on lack of no downtime record for single DM installations. Yeah, that's the thing.

A

Alright, so we have three on but a fault, the issues charts.

A

You go through this in detail here, but my guess is that we have probably.

B

Need to clean those up.

A

Ones open by you prom need bus and ones open by Camille for everyone which includes the references to okay. So it's like a meta issue. It's really that book wish it wishes. The third one down there.

A

Is there any toilet sure that we need for those to make sure they're to make sure we know how the rods going?

A

Like imagine, Monica opted in for some time once well. It's already opted in right now, but right now it's sort of like opt in but looks like a whole bunch of Astrix around it, like it's experimental.

A

So what we'll want to do is likely remove the experimental status right and have more confident foot forward once we have the documentation done, which is saying: hey, try this it, you, like, you, will see good results and you will like it and we're making it require and we're making it the default in X.

A

Maybe it's 13.0 and I. Don't think we need to make I, don't even need to make the default switch. The third two notes it's not really breaking. The only breaking aspect would be the single node restart x, right, they've, led to downtime during reconfigure.

A

So basically, we.

B

Looked at the lemon.

A

Tree would lose a lot of a roll out of opt out dropped in which is basically remove exponential status and stronger nudge and release post right and set a time frame or opt out and then out and then finally removal of unicorn at some point in time. I. Imagine that's a goal. Is it you.

B

Wanted support unicorn.

A

Forever, do we need to.

B

A

Acquitting that forever file.

B

It under stupid question I.

A

Mean it might not be I mean I, don't know like we might have a need for it in some other places, I don't I, don't know like I I'm, not sure those unicorns like psychic have unicorn in it somewhere. I.

B

Don't know I mean if someone really wanted unicorn mic features, they could always tune. Pull me down unicorn equivalent, so yeah. It makes sense to remove it at a certain point. So I retract my question, yeah.

A

Remove the unicorn probably needs to be in a major release like be it 13800 fortunate, oh, maybe not it's not breaking kind of accept the fact that you get the deer get the single, no downtime. If.

B

We want to limit REE from customers, I think their team is probably too fast. We're removal of unicorn actually need to learn a tree cuz well. Well, even any kind of feedback right I mean 13.0 is not that far away. Oh yeah.

A

No I think removal unicorns to the aggressive 430. No though so we can remove that. So let me show you be interesting. If you, if you're a little more concerned on the rollout like in the post, is 11 rollout it's like, should we have some idea of like before you make it opt out how many people actually took the plunge and and if any mm-hmm.

A

So it's sort of interesting on the puma side, so you get a sense of how many people have actually tried this and haven't reported problems before we go and making it opt out and like a little comfort factor. Since we hit really don't have a way to do like a progressive deployment across our like self-managed plead.

A

Unless we need to- and it basically I think I think lemon tree would really be, you know, is it on or off you know, puma or unicorn. Alright,.

A

I'm not sure it's that helpful. It much works that risky in some ways or is it I, don't know.

A

Maybe we add, it's pretty lightweight like I, don't think it's free can we can we? Can we detect that from our color base, like from the rails code or the reading, puma or unicorn.

B

Yes, because of the rugged patch notification right, if we're running Puma with a less than desirable, can fit with rugged, then you get a notice, so I would imagine somewhere in that code, we're doing some kind of form. A detection.

A

Yeah, that's the other hunts, the other facet here on.

A

So the really breaking things are like things we need watch out for our rugged.

A

Upgrade I'm gonna call them bumps concerns, but whatever called them concerns right now, so they on a rugged and the single node reconfigure outage right. Yep.

A

Still have to update to dress, though, is Right. The reconfigure outage.

A

Basically is my lesson packable for single nodes, because it kind of already go down for a short period of time. Right just won't agree if we change like a if we really bump Ruby I think it actually goes down for a period of time. For example, I believe if we bump unicorn like the version since the whole container has to restart but that'll happen. More often now I think the bigger challenge is on a che installations. People might not have the health check setup right, so it might not.

A

It might not Brown out, you might get pulled out of the load balancer. While I was root. Balls reconfiguring, you know, I, don't know. Do we do that? Have we you know if we've done anything they're.

A

Right, let's check with him for there, because we kind of stinky, if you did it upgrade and you had like percentage errors across the fleet, all the reconfigures were rolling that wouldn't be fun.

A

Don't trip her own load balancers, so it's really kind of difficult. So the best thing you could do is probably try based like strong recommend that they have the right liveness check, right, I,.

A

Know I'm just I'm just like brain dumping here, so if this is not helpful on what we need to do to what you're looking forward to get outta here, no.

B

No that clarifies some, like seems like there's still some work in Puma that we can do next so for 12:10, yeah.

A

And then the rugged stuff I, don't we don't want to support it? Right is the general path and if and if you do need it.

A

You go back to work or X thread, one right which is which is the fallback solution which is like the unicorn behavior, and we, when we detect it to some degree me I'm, really present the warnings, which is good.

A

There might be some support. Enable meant we have to do here. Make sure like in Docs like this is clear. Mm-Hm and the support team is aware. So if they see someone upgrading to whatever version make it I'm gonna fault and they're like oh something's, not working, it's like Oh, have you tried logging as an admin? Do you see a blue stump that comes up and tells you that you're doing right, you're running rugged and because, if so, you're gonna need to edit your configuration yep.

A

All right so I think I think that's the thing. That's the accident right. We should investigate the single node reconfigure. We should make sure we have the right documentation in place for the rugged stuff and then the rollout really is just become probably I mean even now they can probably remove the extra medical status.

A

If we wanted to right are we are we comfortable doing that and called I can make changes that release those and just just add a Puma item and just be like you know: Puma no longer experimental, now, I just think it's a ga or whatever Puma Puma fication server.

A

Generally available right won't be I'm. Gonna find them in the text would be like you know the AMA defaults, whatever you can soon or something yeah.

B

Yeah we can update the release post. I was thinking about that. This morning you can include the announcement or a link to us being out on comm yeah I.

A

B

Some more confidence in our from our customers, yeah.

A

Do we own a block, I think, there's a kinetic like today, great recommendation: I'm a blog post on this right. Mm-Hmm I probably can't write that I'm not sure if you or Camille or Matias or anyone wants to I'm happy to review it or write some the boilerplate stuff, but I PI can't be the primary content generation. I. Think.

B

Yes, we can get a blog post, I think largely be Camille and his expertise. That could certainly get an outline from this one I think most of the content is gonna, be here I mean we can call back to when we first started build up and even include some of the results. So, yes, I will put that it seems like a good one for.

B

If not, this milestone the next milestone, probably next milestone well I, guess how important is the timing on this I? Don't.

A

I'm looking at our documentation for Puma and unless we can get I think unless we could beats it up and really make it robust in the next two days, I'm not sure I want to have the big announcement encourage. We go to use it because, like nothing is in here, there's.

B

A lot of information, it's just all over the place so and I, don't think nobody's dedicated to it right now. So let's wait and we'll get some good documentation and we can include it in the release post for 12:9 yeah.

A

I think that's I think that's. Why is because, like I searched, get them in the documentation and they landed on the Omnibus page I think if I can share my screen, so I'm gonna show you what I'm referring to but like.

A

So if you search for, if you search for Puma, you know get lab, come the documentation, you'll land here, which is great. But then you end up at this page, which doesn't really tell you anything. Mm-Hmm doesn't turn on and tells you how to like, like I, just the the worker counts and the thread counts, but doesn't.

B

Give you any guidance.

A

On what to do- and you can click on more deep but come the documentation, it takes you literally to the proof, documentation.

A

B

That's not yeah.

A

That's not super helpful. This is better actually because we actually have the like the rugged stuff in here. Look I want to have Russell, do a pass here, but again, I think we, you know I, think we need more of more like more context like recommendations on thread, counts and work of counsin yeah. Otherwise, it's just I think we might be setting people up for some degree of failure and that's not what we want now.

B

Like I said, all the information exists, it's just spread throughout issues in that epoch, so we have thread recommendations just out of the box and we can certainly add some anecdotal information that we had from calm and the performance issues that we were seeing from that tuning issue.

A

All right so roll out I would say 12.9. Maybe right, let's say that's aim for that: opt-in, opt-out, I.

B

A

Like it's opt-in right now, but but basically like we'll make sure the documentation is there remove the experimental status we'll say it's GA, all that kind of good stuff. Until that night I doubt we can aim for I, don't know 3900, maybe yeah seems like the best place to do it. Well, they will be changing two major things. At the same time, twist goes 11 and in Puma, so I mean sort of multivariable calculus on any kind of problems. That pop up. Is it too much? It suppose.

B

A

Don't know we change both.

A

And then they can schedule removal, probably in 14.0, that seems like a reasonable timeframe. Two people a year to move over if they want to no I, don't think the super urgency here and it is and is that's not really breaking but but if you're on a rugged, you'll have to do something different right, you'll have to like edit document editing configuration because the defaults probably won't have very kind of one.

A

So if we opt you win, you will have to opt out or do something, which is why it's in 13. So we can actually do this sooner. I. Think because really this is this is where the actions required and 13 Auto, May, aggressive, I, don't know.

A

But waiting until fortunate Oh seems wrong and having a minor upgrade that requires some of our largest customers who use rugged to manually edit. The configuration also seems wrong that feels more like a breaking change than a patch release. If you have it, you're like required to edit the configuration to make sure your instance doesn't fall over.

B

Always push it back. Thirteen three to work out some of the kinks.

A

Dropped out yeah.

B

A

I hear you but I kind of feel like if you're gonna require it. If we're gonna make people if people on rugged have to make a configuration change, mm-hmm with or if than if they don't their instance, we'll have negative performance impacts. Then that's probably like a major change to me. hmm I think it's not I mean they. Just if it's not a breaking change, we can see, we can aim for it, subdue no dough and be a little aggressive, um especially considering the worst case scenario. You just turn it off.

A

You know you took the flag, it was not working for you. It's an easy fix. They're.

B

Just no switch it back to unicorn you're, not ready.

A

I'll kill her.

B

A

We can we can keep our eye on it.

A

I'd, rather do it in a major release because of the fact that it's again it required will require config change for some of our customers, but yeah, okay, okay cool we can make. We can just sort of say this is optimistic a little bit because just keep an eye on it put llama tree I'm, not sure you have to be interesting.

B

A

I worth it, it's my six one, five minutes right to add something. The easiest thing give it a, but a little bit over-ambitious, but like an hour or two so.

B

The distribution thing where we.

A

Can make a distribution or or memory I think what other one really? What what has to happen is I has to get into the users paying our B file. mmm Nothing.

A

Barbie I think.

A

A

Where that it's a library somewhere and get lab, it's like I guess I index very well anyways, it's really quite simple.

A

As this I can't it was actually like all right, I'll just trying to find it really fast. I think it's under Lib was version check out. That version shot.

A

Anyways, it's somewhere in here it's relatively easy to deal with.

A

So whatever alright I can't find it moving on I'll make an issue for it. We can discuss where they're going to do it. It's it's fairly trivial, you've heard of sequel, query into that library, file or whatever it's actually want to use, and just basically like we're trying to counter like 0 or 1 if it's on or off, but that's like the extent of it.

A

Okay, so cool performance will recognize from the wrap-up work yeah.

B

It looks like there's gonna, be some work there yep.

A

I think we'll have some work for 12:9 for sure well, yeah, since before 9:00 I'm, some maybe for 1210, we can take a look at.

B

A

We need to get this dog, this stuff documented for sure, I, think in here mmm and.

B

A

Infrastructure, how they're doing it and then yeah and then so, there's this and I think there's we doing some of the additional profiling this just released. There was.

B

A

B

Profiling that was happening, but actually the import team took those took that issue. Okay, get written there's so, but the Ruby on Rails performance training is actually going to happen, starting Monday, and there might be something that pops up in that training that we can incorporate into twelve ten twelve ten is gonna include contribute, so it's gonna be a shorter milestone anyway. So.

A

Yeah, we can also help on any cable. That sounds like a supply going to proceed, so we I'm not sure we again I, don't think we should be the one to implement it because we don't know their use case requirements, but we might be able to assist with some questions. I think there is I thought about going to action. Cable. First, where scares me I.

B

A

Know how you turn the bombing get live without calm, without setting up like additional nodes.

A

And then I'll think maybe and then I think the other thing was just in the profiling work right like um what else is.

A

Where else we next optimize on some of these things, yeah.

B

We have some older issues. I have some ideas on where we can. Let go what toes to you so I'll dig those back up and throw them into our next planning issue. Clones tracing.

A

Okay, I'm speaking of profiling, if kind of feels like if we get if we get like a trait like a elastic tracer on or really any kind of tracing on, get lab that might help us understand more of where these bottlenecks are I.

A

I know we have added support for tracing by we be so-called experimental, which also means.

B

A

Customers don't run it generally and so until we kind of feel like we can run it at scale and go calm, I wonder like I like who would do that.

A

That's one thing that doesn't fall anywhere kind of message: why why it's not done yet because last contribute I'm, pretty sure Andrew had racing so under head tracing support built in the gay lab and it's been a year and we haven't gone anywhere Ethan. Oh it's just so sad, yeah.

B

Valuable, we should thank you so yeah.

A

All right, cuz, you can kind of exactly see where this like, where the, uh where all the time is going in these requests, so all right, I'm, not sure if it's memory but I, don't know well. Who else have worked on this stuff? There's a broader question: the product team around like technical debt and making sure we prioritize it. A lot of this stuff doesn't fit anywhere yeah. It's the same time with, like our upload, our called storage upload frameworks.

A

We have to backer an uploaded, a direct upload and who maintains those who's, working, try and cut us over from one to the other, and no one is busy with the answer. So as far as I know, it's just my eyes area. So it's like I, don't know. That's one of those cross stage frameworks, that's used by multiple stages who owns it. I, don't know like no one yeah.

B

Other companies I've worked at I've had like a core team, their own stuff, like that yeah.

A

And so I don't depending on whether we find like what we find in the process of doing some, the profiling would be a real performance training yep.

A

Maybe we picked movie, take some of these things to help further like investigate, what's happening and also just level up our ability to understand, what's happening in our in our application.

B

Yep I mean worst case. We get in foreign teams like hey. You've got a mop on that carriers identified, send them some tech, debt yeah.

A

For sure, so please I think that's also. My thing is some of the charter of the team wishes to try and make sure in the tools and process tools and frameworks in place that to determine when these things yeah, okay,.

B

Cool so I think that is.

A

That sounds good, so I'll work through this list and trying.

B

A

Sure we have issues for all these things and milestones attached and then also we PI to make sure we double. We look into this one with the infrastructure but cool, and then we and then we have someone to work with Russell on the documentation, right, I. Think because.

B

A

Name estate, I.

B

Can I can point him to where, like, if he's gonna, take the lead on this and start writing up the documentation? I can point him to where a lot of this information is so, if there's nobody else, I can help them all right. Cool.

A

Awesome: okay! Thank you, sir I'll uh I'll work through these cool all right. So that's how yep.