GitLab Package Group, 18 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Stan Hu / Hayley Swimelar Container Registry

Description

A discussion on topics around GitLab's Fork of Docker's Container Registry.

A

A

Yeah, so just immobilize so like they have a place to put documentation that we generate for this project and like where is the best place for that yeah.

B

Obviously, I think the best place is in the repository guys, because that's what proof we're gonna look, you know we can either put it in the readme just edit. The read me directly or have our own specific one about our notes. So I don't know what you want to call it maybe get lab specifically me, like you, love like read me, get lab or something just make it clear that these are our notes.

A

Yeah and I think that's been like my my question: what is the DFB assisting doc here got their documentation, which is not, which is pretty valuable, I've read you some of it and they talk about. You know like the payrolls of online garbage collection and such so. You have there's good documentation. There yeah.

B

What should we do about that? I'd actually dump a good answer for that right now, I, actually I haven't looked at the documentation and what you know where we went my edit. Is it actually part of the repository or some separate repository.

A

Yes, there are some.

A

Stuff, it's like discussing, like you know like this, is like, if you want to like, do a merge requests for all and garbage collection like please consider you know, regret consistency and.

C

Just like the readme.

A

And I believe there's an API documentation so like, as that changes I'm, not sure like it seems, like part of me, wants to say I think, there's a reason to have like a historical like document that's like like upstream and.

B

A

Annoying like as upstream changes of documentation to have like conflicts based off that right. Oh.

B

A

B

Question I, don't have a great answer to the I said. You know, I think you know, let's take a step back and look at the I. Don't know if you read the issues where we sort of talked about the container registry. I think didn't our and but basically this issue that Aaron linked at sixty-five that one. This is sort of the initial discussion of here all the things we want to do with the container registry.

B

Here's the approaches we could go and I think for King was wasn't one of them, but there was other discussion about people's. You know, I, think, Camille and and Demetria I think we're proponents of just writing it and creating a whole new registry ourselves right because.

A

I actually think.

B

It was going to be all that much work. I was more hesitant to that, because I know that registry has been in use today quite and we use you know we depend on it today and I I took I thought it might be a little bit more straightforward to fork it, and if we got to the point where we decided that forking it was just not worth the effort, then we could go, pursue the rewriting it. You know we decided that we've gotta totally we architected.

B

So in some ways this is a learning process right now to figure out. You know, as we maintain this registry and we find that we want to do all these different things. Is it just architectural II? Not what we want and we want to actually do go with the rewrite. So I think right now we're just trying to figure that out. Okay,.

A

B

Guinea pigs to figure out like what is it registration or code doing? How do we make changes, update it and so from what I see so far is the code is pretty well architected, so yeah.

C

B

Like if they feeling like, we need to go off and rewrite everything, but then again if it increases our velocity and we have more people who can help out if we write it then maybe that's a reason to do it, but as far as I can tell it's still, the project is still chugging along they'll, be it slowly.

B

We will have to kind of figure this out and all the questions you have right now are are really. The heart of the question is: like is a fork? Gonna be workable right now, we're are. We gonna have to make so many changes in the documentation that it's you know our fork is so different. Now that really we can just treat the whole documentation as our own like starting point, and we can totally revamp it as we see fit.

A

B

Know I haven't done it, we have I, don't think either resident done anything that warrants a huge documentation change. Yet no I saw I think as we get to that point. We can kind of figure out. You know what do we need to do? What do we need to update? Can we upstream this? Can we get their attention? If we can't, then you know we may just have to own everything, and just you know, yeah.

A

That's right, that's what I've seen with docker upstream is. There just seems to be PRS and issues that, like they get so far, and then they just sort of like break the radio silence. There's one I like whitelisting of media types like there's no media types and.

B

A

Like why don't you not make it reg X is just make it plain, listen strings! It's like great I'll. Do that it's.

B

Probably a but you know those are open source contributors right, so you can that.

A

B

All the time right.

A

B

Docker, employee, that's a whole different question.

A

B

I mean the least we can do is try to incorporate the best ideas and try to you know show that we're moving quickly, be ironic. If that you personally could just suddenly caused the entire docker team to realize that hey get labs moving faster than we will have to see, certainly feels like that. Right now that you're making more changes and.

B

A

Yeah I saw that just now haven't really got a chance to read it. I, don't.

B

Know what's going on so anyway, it seems like the whole world is moving into like an open-source. Docker container so doesn't feel like there is a real, like business value, having a closed source container registry. So you know quai open sourced their source code yesterday. So.

B

So, like I said we're gonna figure this out and I. Think these are questions we can cross. We can cross these bridges as we come to them. I, don't think we need to solve them right away, but, okay, what's your next question is about general direction, so it's the same idea right is I think you know our scope is so large. That is, unless we get buy-in from docker to do all this stuff.

B

I think we're just gonna have to take this registry and run with it, because it feels like all the things we are doing. Our fundamental changes in the registry right adding database support all that.

A

B

That is going to be really really a big breaking change and I. Don't think we're gonna be able to support upstream if.

A

They're not gonna help us so.

B

I kind of work it you know the kind of tea to pass, one of which is like we make all these changes we upstream them, they adopt them very quickly. Second, one is like they. You know. We just make all these pages and we become the de-facto container registry like go Arbor or something like that right. So.

B

Yeah, so it's it's, what kind of being pioneers here and I? Don't think we have any answers like I've said I for you, Dan email from the previous CTO docker, yeah I think he obviously wants to avoid a fork as much as possible. Right look at the things we're trying to do it's like way out of the scope of what that current.

B

You know unless it's basically what Daka Enterprise is doing I think so there may be some things internally that prevent us from working with so I I think we should just do what's the best interest of gitlab and make it open source. Suddenly we get a lot of traction because it made a lot of progress on it. Then great. So it sounds like like everything we do.

B

I get loves and we got to be incremental here, so you know to make changes and then, as we see like the knee, we may need to kind of revisit these kind of questions, but I think you're, asking all the right questions. Just like we haven't crossed the room right now to say that way. We're gonna have to completely diverge I. Just think we diverged significantly right.

A

Yeah, so just I guess for right now: I guess the consideration is.

A

Would we be hesitant to pursue a change that would change the architecture because it does make it more difficult to to take any changes from those dream. I.

B

Don't think we should worry about that at this point that we should do like if we, if we need to support online garbage collection, let's do what's the fastest thing for us to make that happen. If we need to upstream that or if they want to adopt that, we can put people on it later. If it's a, if there's a strong case for that I, you know, I, like I, said: I, don't see that happening right now.

B

So, let's just let's, let's, let's revisit that, if that ever comes up again, but I think we should just be quick move quickly and do what's best for supporting customers and users and the fork is really just a means to bootstrap like these changes as quickly as possible right. So it's not so much so we can upstream as much as we can, but if it's like I've been trying to upstream my changes but I just get zero comments on my polar quest. So there's not much.

B

We can do if there's no and no one reviewing the code that we're submitting.

A

B

Know, ideally, we keep whatever we code we can easily up. Streamable like the changes, are very clear, they're very portable right. You know it does take work to upstream it because we've made you know. They've they've made some changes in like the go modules dependencies and you know the 2.7 branch doesn't have that, and so it's just you know it does get a little complicated with dependencies and all that, but yeah I still feel like we look. Let's just do what we think is necessary for internally and if we get more buy-in, let's we can.

B

We can certainly change our course. Sir. Okay.

A

Yeah so yeah yeah yeah, yes, I I, feel like I've, been relying in you a lot for code review and merging. Is there someone else like just you seem busy yeah.

B

With the current registry, I'm not sure how many other people internally have had a reason to look at this source code and the TAS, and all that yeah. It was probably the person who has he's also busy guy so I, wouldn't want to say Camille's also maintainer, but I think it makes a lot of sense. To put you know, another person or two to be able to review these. These changes, I, don't know who that could be.

B

Obviously somebody else on your team is be the most likely candidates, because you're trying to quickly in the registry.

A

We're adding a person should be adding a person, I'm, not sure I'm, still I'm still relatively new.

A

B

My understanding there are people who can review go code here internally, so maybe the short term we just assigned somebody who has some go experience like I said Camille made, probably the most likely candidate see how he understand the registry really well. I would nominate him as like kind of interim until we have more people and then we can propose like if you, if you find that you need another reviewer.

B

Maybe we should expand that to some of people in the runners runner team, because they they know they probably have experience with the registry as well and other people internally. Just interested in reviewing go right.

A

So just be the best way to contact those people like I, know with all the Ruby stuff there's like review or let which seems really convenient, but.

C

A

Go, it seems yeah yeah.

B

So, like I said, I think we should propose. Maybe a lightweight container registry maintainer program just put something out there and either like a merge request.

B

As just as you know, if you'd like to be involved with reviewing go or container registry, you know Mike yourself as a you know, I think I, don't know, you've seen this engineering projects page on the handbook, but basically people can tag themselves as, like you know, back-end, reviewers or back-end maintainer I think we should do something like that for for the registry, so that kind of Institute you know volunteers and also people weren't formally assigned so.

B

There's an injury project page. You should add the container register to that. We should add maintainer zan viewer. You can add me as a maintainer for now and then probably you and I as a maintainer to start and then you know we can nominate reviewers. You know just through a merger quest, we can just say: hey Camille you're nominated as a container.

A

B

A

That makes total sense so that any parts of the code that you want to point out to me in particular that have relevant surges.

B

Nothing on the top of my head, like I, said. My exposure to a lot of this stuff is just in the context of understanding how stuff gets stored in object. Storage. That's really what I've been doing the last month. Like understanding you know how does a manifest get updated? How does the driver get all your you know? Work is directly relevant because I touch that stuff.

B

There's a lot of other stuff I haven't looked at, so it may be useful for us to start documenting like what we've learned because I don't know if there's like a development documentation around the doctor registry right now, and that may be another way for us to kind of get more contributors because people say hey, look it laughs. As you know, this great documentation, I, don't understand the registry.

A

Right yeah I mean I've got some like it'd, be nice to write down like the manual testing process for the Mineo, because, like there are some like it's, you know, you've.

B

A

A you know: I figured it out, so you might as well have so do like not have to like somebody else. You.

B

A

B

Exactly and I think that's probably first place, which is so. We should probably start development documentation and just down today, I learned kind of stuff. This is right.

A

Yeah totally makes.

B

A

So, what's the deployment story like how do I, you know we have issues and stuff and I should suppose to mark like hey. This is a production now like how do I know when that happens, right.

B

Well, I think: first, we said you know we should document like the the Omnibus and the I think I linked in one of the issues. You know the Omnibus file that you need to change most of the registry, the the cloud native images for the registry as well.

B

We should document that, but then how do I think it is? Your question is like how how do I know what version has actually been deployed? Yeah.

A

How do I know you know what version is running in production and yeah.

C

A

Just so I can like keep on top of that, so I can report back to my team like hey. This issue is like it's in production now, like that's.

B

A good question so the way it's done right now with get labs heeyyyy heeyy is there is a there is a I, don't have been in the announcements channels, but there's announcements when certain packages get deployed and it's it's there's a it said: there's a version tag and like a shot on that, like it's, not a very.

C

B

Way of knowing like exactly which container registry is being deployed.

B

Either I mean, ideally, we have some sort of page where we can look at and stuff showing. What version is is is running right now we probably should open an infrastructure issue, because right now, the way it works is that the registry is running in a container on kubernetes and in the past month of the way I. What version we've been actually running, as I had to look I actually had to log into the pod and actually run the registry version, which is not the right way.

B

Nobody wants to do that, so we really want. You guys have some way to report in like Prometheus or something urgent is actually running. Is there Prometheus metric for version.

B

There are these metrics in the docker registry. I think there are I, think.

A

There's a couple maybe.

B

The first thing we do is, we add metrics for like what version are you if there's not there and then scrape them, and then you can have either the dashboard or Prometheus like a query that just tells you exactly what's running so I think that's the way to do it.

A

Yeah so I'm so for bigger changes so like if we do online garbage collection, like obviously.

A

Some pretty heavy testing before it gets deployed in production. Is there like a state I, don't believe, there's a staging environment or canary environment for the registry.

B

We're staging yeah, that's how they tested out the whole Cabrini. Is we have a staging okay.

C

B

We do have a staging and pre-production and Deb's out, get labelled. Org also uses register registry as well. So we do have a bunch of places where we use it. So the testing part of the infrastructure side is there.

A

Yeah, there's today is that something like you know if we have like front-end jobs that are making like for API calls to the registry. Should we just like is that a goal to just like have like are like extending API or have our own version of the API in point we can hit to like sort of us in five specific workflows, yeah.

B

It's a good question: I mean obviously the docker engine ape I. Guess the question is your tell you how to talk about the docker engine API, the basically the thing that interacts with the docker command line or you talking about like hey. We want to actually query. You know this in a certain way and you know, even though it's not supported by that you know the command line. We still want this data.

C

B

Command line, what is interesting, because, obviously we can't make any changes that will break the command line. Cuz, okay, well,.

A

B

Think that would break a lot of things. Yeah.

A

I mean it would just we have to.

B

I would think that we want to extend it with a different API or a different version, or something very obvious that this is. You know our extension, so I would say yeah, let's where we can just keep the existing it guys it's possible with normal clients. We may again, we may have to revisit that. We may decide hey look, you know we really need, but it seems like shipping, your own command line. It's going to be, you know a nightmare, so yeah avoid that as much as possible. I.

B

Agree, we should probably just say like: what's not, we had to have a really good reason to free think we have one yet right like if we need, for example, if we want to support like hey, let's, we need an API to trigger garbage collection right. That's gonna be a specific thing that we create like it may not even be a docker specific engine API. It may be like a good lab API. That says you know to trigger this garbage collection, and then you know.

A

B

That's yeah, mostly the things like that.

A

B

A

Less specific work close to it like have it in points for that right, cuz.

B

I can see we're gonna need to probably a database support to our registry right, like suddenly we're gonna have to track all the blobs in a database and not on the file system. Seems that seems like it's a given now, given that like this is Harper does this is what Quay does? This is what the only way you can possibly do a do tracking in a sensible way.

B

You can't have it done in the file system, because well can you I, don't think so, because you need a database, but maybe maybe I'm wrong, because kids do this right.

A

B

It also doesn't work with object stores. Yet right, like we're talking like in its assumes everything's, local and right, walking, walking the file system and get is, you know, is normal, but doing that in object. Storage is insane right right.

A

Yeah and that's that's one of the things I've been thinking about with the issue to like improve the speed of the garbage, collect command. Mm-Hmm and that's you know, I haven't done that ever done the testing yet, but my assumption is that this is like that's mostly going to be storage back independent like almost, if not eighty percent. More so like it's one of those things like if we do eventually switch to online garbage collection- and you know, doesn't even work like trying to speed it up because it's gonna be different right.

B

How do you do it at scale like well? First of all, you do that scale right if you've got like you know, you got 300 terrible, petabyte I think we got in some petabytes of data in our registry right now, like good doing it. The way we do now is impossible right. You can't possibly scan every blob and figure out which ones aren't reach of all which ones are nah. It's just like. We create more data than we can't scan it right so like this will never work unless you've tracked this information.

B

As you go right, so you need some sort of database whether that database is stored in like an old database or.

C

B

Like in some other you know, back-end storage is another question, but to me it seems like the database is the most sensible way, because you can do all sorts of querying around actually figure out. What's not reachable with relatively simple reference, counting yeah.

A

I mean that's. That was my first ideas like well, that's you know you need something that has like transactions a we just.

B

A

Can like actually account for that data and not have like that yeah? The fact it's kind of it was very surprising when I realized it's just it's just like it's just flat file, so I was like kind of taken aback right me.

B

Too, I was kind of surprised that, like this is this is what we're shipping to it, because it's like a really a simple and there's lots of issues with it. So yeah.

A

B

And you know when.

A

B

Sometimes the the best problem yeah you have to solve yourselves. Therefore,.

A

B

A

B

Yeah all the right questions like I said this is pretty new ground, so you know there's none of this is set in stone right like if we decided I. You know, I know that when we first discussed about what's the direction we should take, I said we should probably look at go Harper really closely because they're doing a lot of stuff that we want and then I think we just sort of dismissed out of hand that, like a the old stuff, is too hard to strip out or it's to like, but I'm, not sure.

B

If anyone actually evaluated like hey they've got all this great code that we can just pull out for you know database all that, so I do still think it's worth spending a little time making sure that, like going down this Fork path, is you know someone that we were not reinventing the wheel? In other words, like there isn't already code out there that we can leverage to get us to this?

B

You know to the right point: faster I think we just I think you can talk to Daniel, though I think I think we decided to go with the whole fork because it did seem like the most. You know incremental step. We could take the small stuff. You could think because we're using it today, you know augmenting. It seems like the least risky thing, because you know we take on a go harbour today. It's completely different code. It's completely, we don't know if it's gonna be compatible with what we're doing so.

B

There's a lot of risk there, but it may be still useful to kind of look over their shoulder to make sure that you know. Hey.

B

We've already solved a lot of things that we're going to you're you're the first person on this team to kind of focus on this, which is great so I hope you enjoy these kind of like open-ended.

A

B

Think you have a lot of you know. I think you have a lot of sway into like how we kind of move forward here, because if you say.

B

Rewrite or just go with, you know different registry, then I think we can set in stone. It's mostly. This is the approach we're going now, let's revisit that, but as far as I can tell right now, we need to support the registry we have today. Okay, that's what we're doing yeah.

A

That makes that makes sense. A yes I've got a feeling, it's more exploratory, that I had like I figured that you guys like doing the fork, but I gave it an exploratory. Now, yes,.

B

We're also fixing things that are critical like right, so we got a we've, got a fix, the things that are hurting us, but we also have to a longer-term picture of these- are losers. 100, none of the things we want to add in the registry, so right.

A

It's an entire application, yeah.

A

B

Is all your team will be growing in the next.

A

Pretty good idea what to do going forward at least yeah I appreciate appreciate, okay,.

B

A

All your help now feedback sure glad.

B

You're jumping on this Greg glad to see somebody else. Looking at this I was actually kind of surprised. Somebody was somebody came. It started so many mirja quests for those, because it's like this whole project really just started in the last month or so yeah.

A

Yeah happy to be here happy to help out all.

C

Right nice, talking to you.