GitLab Scalability Team, 17 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Discussion around Platform direction, Scalability:Frameworks and "Pajamas" for Infrastructure

Description

Liam and Marin discuss creation of self-serving platform in Infrastructure and how this aligns with existing design platform

A

Okay, perfect so um marion. I was keen to find out a bit more of your in regards to your vision for kind of this pajamas for infrastructure concept that we've spoken about.

A

When I joined a team, you spoke about kind of having a contained box um which is kind of like a set of tools and maybe like procedures, and that we can kind of like develop ourselves within scalability to bring value to kind of other other parts of the business, and I shared some of those ideas with the team already, and I think there was a slightly of confusion with pajamas for infrastructure as a concept, I think some people aren't too familiar with what pajamas is from a ux kind of standpoint.

A

Maybe they haven't interacted with the tour, maybe even if they have don't quite understand how that would translate to something that we could build within scalability. To kind of, I guess, help us and the rest of the business scale.

B

Yeah, that's a good call out. I didn't really anticipate um that you know it's not that familiar uh for for engineers in infra.

B

Now in hindsight, it makes sense right, like they haven't, worked with this directly, but I was kind of assuming that if you hear pajamas, it's a very catchy name, so you'll kind of at least try to see what it is, um and I mean what I can do is uh I can share the screen and show a bit of uh what pajamas is and how this actually relates to to where I want us to consider going next and where I want us to see uh an improvement uh in on the infrastructure side and I'll try to kind of tie this in with some of the other things that we've done, that are pretty much aligned with that.

B

We just don't have a fancy website like uh ux team has, and obviously they have like two to three years uh head start on this. So of course it's like very much more organized. So let me start by doing that. First and.

A

I think examples of things that we've already done, which could kind of fit in there will be really helpful as well to kind of live um some examples so yeah. This is the.

B

The pajamas uh system- I don't know exactly how they uh called it, but uh you can see what it is here, so it's design gitlab.com. So the important part here is that this is not just um a set of documentation that exists here. Right like it's not just you know. um This is how you do things. This is not a duplic duplicate of a documentation website. This is actually an extension of it. So in the docs website, docs website, you have certain uh guidelines in what a front-end engineer.

B

Developer contributor should do when using some of the design components, but it doesn't go into details because right, like that's, not the right place to do that in the development docs. You want to just explain. This is what you need to use if you need to know more details go here. I also understand why this is not in the handbook, because this is very comprehensive. This is not something that is on the company level. It's it's. uh It completely makes sense that it's a thing of its own.

B

So if you take a look at that website, like I said it is about some principles that exist and um some guidelines on how to think about certain things.

B

But what I think is very interesting for us in infrastructure. Is we're not really that far away from this, because if you think about putting something in production right like making it run in production, we are no longer just going to you know ship a feature and hope for the best we have set of items in the definition of done. That needs to be considered.

B

Something recent, for example, that is still active, is rate limiting right, like we, a scalability team, did rate limited in project almost a year a year and a bit ago, almost a year and a half ago, and that you know gave us a step forward, but the product has grown since then, further new items were added and, as new events happen from the outside, we realized there are more items that need to be done.

B

So are we a scalability always going to go in and you know, do a project that will add a new component to the rate limits, or are we going to let development teams who are supposed to be shipping product features that elevate the business further? uh Do rapid actions regularly to ensure that they try as everything that happens in there?

B

Well, maybe, but I am trying to say that maybe we want to make this better for all of us and I think scalability is positioned the best to do this, especially as part of the you know next iteration of the platform uh section, so you know just to tie this back in um to uh to to pajamas or the design website.

B

You know, apart from having this as a set of guidelines, it also has certain um not only instructions on how to use certain things, but also has actual items that you can use like. I don't know how designers call this, but let's say, like I open this label component.

B

This is literally a guideline in how someone adds a label into on any page of gitlab right, but it's not only a guideline. It's not only when to use it, how to use it and so on. If you open this, like you literally, have a component that you can play with edit it out and get that object, put it inside of your application and that creates a unified um user experience across our website.

B

So um I I don't really recall how much of this you remember, but only a couple of years ago, before we had the the design gitlab com, people were using different, drop-downs everywhere right, like whatever was available at the time. Whatever was the shortest.

B

This type of system unified this across, so they didn't go and replace every single button that exists out there. They set a set of guidelines. This is what you need to do going forward.

B

Every time you go through a page and uh update it, you need to follow the new guidelines, so this is pretty much what I want to see us set up in infrastructure and specifically in scalability platforms, and I actually have an example of us already doing this just on a very lower uh level scale, without a fancy website and so on, uh which is lapkit.

B

So this is created by andrew and lab kit is literally this. So we have two lab kit repositories, one for ruby and one for golang, because we do have do those two types of services and look every time you want to import a logging library into your golang service. This is what you need to do. You don't have to think about how the structure needs to look like and so on.

B

You import it and then follow some of the guidelines that exist there on how to utilize it rather than you having to adapt it or change things, because we know that once you actually import this and utilize it in the way that we set it up, you don't have to do anything else for us to put this in production right like we know exactly how to plug this into the other interfaces that we have and we actually get out of the box solution that developers don't need to think about and, more importantly, we in infrastructure don't actually have to do much more to put in production.

B

So if I kind of go back to the rate limiting example, how amazing would it be if we had something similar to this, where some every time someone creates an endpoint like an api endpoint or you know like it, doesn't really matter- exposes a new endpoint if they could do an import similar to this and say all right now, I can go and play a bit with.

B

What's the input all right, like I have, this type of service uh I need is estimate is going to be probably used this way based on whatever the prep work was done there. So I expect it's going to be okay to run um a rate limit of.

B

I don't know, 2 000 requests per a minute, for example, and I'm basing that off of what we have currently in the system on gitlab.com, specifically in a similar type of endpoints right like we can, even whether we visualize it this way or not, that's something long term, but it would be quite amazing if we can. You know if you select in our case it wouldn't be background color, it would be. You know graphql endpoint, and that would give you results for a similar endpoints that we found you would go to an existing service.

B

You would see their number of requests per minute or second or whatever, and that would kind of give you an indication with which you can work with. You add that into your into your feature, and that is I'm not going to say it's going to be a minute of work, but it is definitely faster than creating a rapid, rapid action after the fact, or you know, creating a new project that will now have to revise how the uh the feature works and so on.

B

So this is something that we already did before and we are using successfully. So basically, what I'm looking to do with pajamas for our infrastructure is formalize this across the different projects we use in scalability and start working towards this uh infrastructure system.

B

um That would uh that would unlock us as well to support not only development but also the rest of the infrastructure, department teams and even ourselves in some cases,.

A

Amazing yeah, I I kind of understand that I think it's a nice vision to help.

A

I guess I've seen scalability, but certainly in wider infrastructure as well, enable other parts of the sort of development organization um and to bring, I guess, consistency and hopefully get to kind of like one way of doing things, and one thing: it wasn't entirely clear to me there, and so I I really like the rate limited example.

A

I think the fact that you focused on endpoints because there's a whole bunch of attributes that happen or exist against an endpoint right, there's rate limiting custom targets, for example like understanding what the threshold is or how you set a threshold.

A

Do you see this system potentially being a way to just get guidance of how these things work, or do you see it as being a tool to try and action these things as well? So I'm kind of thinking like the industry standard term for these types of tools is like an internal developer platform. Do you think they're in the in that direction, or not necessarily.

B

Yeah, I am I'm not really sure uh exactly, um and this is why I want to kind of think through this a bit, because there is a bit more tied into it, right that there is organization behind it and so on, but ultimately we as a platform team, will have to move in that direction, because, apart from the growth we are seeing right now like there, there is very rarely uh in a significant investment that a company does into its platform teams right, like they're, looking to get the the teams that create customer facing value um be efficient and they support uh whoever supports them um as much as or other as bare minimum is necessary right like just whatever you need to do so.

B

I actually, I don't really know whether it's gonna go in that direction.

B

What I would like is a different story right like that is what I would like us to see do not only with scalability but also with delivery team uh as well. Eventually, because this is similar story, it's enablement, it's enabling others to do things more efficiently, and then it elevates the work that all of us are doing as well to a different plane.

B

So, instead of resolving the same problem over and over again, you need to think about the different use cases that some of the teams have work with them and um resolve a problem for a lot of different folks. So the answer is, I don't know I hope, but I would like to start at least with something.

B

So if we can start with formalizing the work we've done with lab kit right like that is the first thing we already have something: let's create you know, set of guidelines and say that every time you need to lock something. This is what you need to do.

B

We have some of that written down, but we need to do a campaign, a regular campaign of working with the development teams to emphasize this at the beginning of develop development, rather at the end, and then also figure out how to include that into the day-to-day workflows, whether that's through pipelines or something else. I I honestly don't know and then try to apply similar recipes to existing things, that we have right, like one of the items is rate limits that we need to start talking about.

B

Another item is something that is ongoing, currently, which is object, storage, which is a very similar story to this we have, I think, 13 buckets at the moment. S3 buckets for different types of files that are actually essentially all the same.

B

They are different in the fact that they are serving different features, but they are doing the same thing, so we are actually pushing this burden onto our customers because they need to configure 13 different buckets to get this done. Instead, we could have created well, we couldn't have because we didn't exist at the time, but the point is now: we can set up a set of guidelines in if you need to interact with object, storage. The infrastructure department, scalability team specifically, is setting this guidelines.

B

This is what infrastructure we have can support. We work with the product team to figure out what is our, what our customers are actually using. So we can say- and this is what our customers are using, so these are our ceilings. This is what we can absolutely support. So within that we are telling you that you need to do this these and these things in order to store your file in object storage. So that's another one of the examples and I'm absolutely certain. There are more and more like that. Sidekick comes to mind.

B

Reddish comes to mind like all of those items. How do you interact with parts of the system comes to mind, because that is what makes development more efficient, but also allows infrastructure to support uh development needs.

A

Yeah, I can see a huge amount of scope here. I think something we spoke about recently in the sustainability team and I think it's wider again into scalability, but is a creating a life cycle, um and you just go back to your point of end points and being able to understand the attributes and kind of like how you set things and what they should look like, but also to create that life cycle of for this endpoint. This is how you get visibility as to how it's performing it needs a dashboard that relates to it.

A

This is what's expected. um I think there is a huge amount of scope in terms of giving all that visibility to engineering teams in in one place. um I can certainly see that I I I guess thinking beyond that, and I I guess similar to my previous question around um the internal developer platform. We don't know we don't know what direction this would go in, but it feels like this is just another part of the devops life cycle. Right like it's like part of like the, maybe like the manage or monitor, or it probably fulfills.

A

Quite a lot of those different um different stages: do you think these do you think it's feasible or efficient to try and build some of these things into gitlab itself, or do you think it's like first iterations we should be just looking to get it started somewhere and then see how it grows.

B

What I've successfully done during my tenure here is once I know what the requirements are, what do we need build it back into the product, so have an example of you know working that in in the delivery team, uh specifically where we really didn't.

A

B

All of the items that would be necessary to communicate how a package ends up, or you know how a deployment ends up on.com. We build that outside of uh gitlab as a as a quick sketch, basically put it in motion, got some feedback, and then we built it back into the product. So over time, as we go into um not go into, but running, not even running to develop is the word develop more of the knowledge of what is actually necessary.

B

We can uh back that into the product, because if we are doing exploration inside of the product itself, that creates huge amount of overhead. It's I think in the handbook we say as long as it's within 5x overheads right like it, takes 5x time longer to build this, it's okay to take the time, but I think this is because we don't really know uh all of the items in here.

B

It's probably going to be much much more than 5x, so I think we need to lower the barrier, get the feedback faster and then see how we can actually uh bring that back into the product.

A

Awesome this is exciting. I think it's a it's a good, um a tangible way to kind of show you what the vision is as well for scalability, um it's a good way to expose what we're doing um and, ultimately, it's a good way to help us have like a higher leverage in terms of how we scale other teams and so yeah. I think, there's obviously a lot for us to think about what we can do, but this is uh yeah. This is exciting.

B

Excellent liam thanks for asking that question uh actually- and I think I hope this recording is actually going to help out others and, more importantly, I just want to be clear about this. This is why we have hired some really really smart folks uh in uh scalability. I actually need help and you will need help as well, like all of us will need help to develop this, um and there is no absolutely correct or incorrect way of do of doing it.

B

I just want to make sure that we are at every single step, delivering value, and as long as we are delivering value, we can work to create whatever whether it's an internal development platform or something else uh we can. We can work and we can work through that uh as we go along anyway. Thanks thanks.