GitLab Package Group, 22 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Container Registry Pull Codepath Deep Dive (Filesystem Only)

Description

Overview of the execution path for container registry pulls when using filesystem metadata.

A

If, okay, um you know, welcome everyone to uh container registry deep dive, we're going to look at um just the code path, that'll that you'll take during a push and a pull so we'll likely not cover everything, but we will cover some things all right. um So the biggest thing is: if you have any questions, just ask I'm going too fast. Let me know so I'm going to share my screen and we're just going to look at the push-pull diagram.

A

Okay, so um this is the pull we're gonna do this first, because it's um way simpler and we're gonna we're gonna just skip over some parts of this because they're not as interesting, they're, more complicated in some ways, and it's just easier to ignore it. So we're not going to do any of the auth we're not going to do like the check v2, because it's not interesting we're not going to do the auth, because the auth is its own subject.

A

So, when you pull a container image, you know you want to pull down, there's really two parts of a regular image, so a regular container image, there's the manifest and then there's layers and that's that's kind of the deal, that's kind of the dichotomy. If it's not a manifest it's a layer or bob.

A

um So when you pull you get a manifest by its reference, which is either going to be the tag which is um almost exclusively going to be or the reference.

A

The other reference is the digest, which is the hash of its content, and um then once you get that that's if that checks out everything's, okay, you're also gonna get um the image configuration, which is a special blob.

A

So you always do that and then for each layer, you'll get those um and those are also the same path. They're not treated differently um as far as storage goes, they are, but the configuration is special and we'll get into that, maybe a little bit, but.

A

Typically, the blobs you'll treat as um opaque. You know you don't really know what's in them in the code like they're, just a solid chunk of fights the manifest um we do inspect that quite a bit just because there's there's numerous things that make us need to know. What's what what the manifest is about, um we'll see that more in the pull just or the push, rather just because we need to validate that everything's okay and we need to read the manifest to do that um yeah. So, let's go down.

B

A

B

Yeah, uh so in the first request, there's it's just like simple get to the v2 yeah. Do we specify there that we're gonna do a pool so that we get a reference back.

A

Nope, um that's it I'm glad you asked because that's a that's a point that we that was normal to me, but it's probably it's something that it does come up and that all these requests are not correlated.

A

That's the client decides to put these together correctly for the most part uh and four pulls. You know somebody could just pull blob down and leave us alone for the rest of the day right. um So clients generally coordinate the right way to do this, and this is specifically a doctor client. You know it's, um you know for pull. You know, there's only so much you can do, but for a push it's conceivable, you could do it slightly different, um but yeah. So this is.

A

The client is basically responsible for coordinating how to how to work the api properly. If that makes sense um in the same way that it doesn't say, hey, I'm gonna, I'm gonna. Do this pull or we're gonna. Do this push and like look for these requests coming, we don't really know, especially for a push.

A

You know a client could just push blobs.

A

You know not thinking about life cycle management, but a client could push one blob every day for for a month and then push a manifest six months after that and like we wouldn't we wouldn't know the difference.

B

Yeah get it so. Basically, the the container registry doesn't care about like an image in general. It just knows about blobs, and this manifests kind of.

A

Yeah and that that's that's great, I'm glad you mentioned that too, because, like you'll find in the code, nothing is an image.

A

This there's just plain that mentioned: there's manifests, there's blobs, there's manifests and then there's blobs and that's that's kind of it. There's also tags, but they're tags for manifests so um yeah. That's, that's that's one of the things that is it takes a bit to get used to is that the registry just does not have this um consolidated view of an image everything's very broken apart, and that makes sense in a way that we don't really do anything with the image as itself like, because we would have to make it an image.

A

You'd really have to reconstruct it from the layers and the registry doesn't have any sort of container runtime. It can't do that. You know the registry really doesn't know if a if an image even works right, it doesn't even know if it's like makes sense. When you put it all together, it doesn't have to build it. It doesn't understand.

A

um What's there at all, which is you know, size and downsides? It does simplify things a bit, but it does also mean um you know. Certain things are, are harder to do or just aren't available, but um yeah and that'll become clear as you work it's hard to like you know, you'll get you'll get a sense of that as we go through, but I also, but so just.

B

I suppose one of the problems is, you can end up having a lot of garbage in there like a lot of blobs that are not even related to each other, or you know if whatever client decides to push yeah sure of.

A

Course, and um as you read, the garbage collector um um online garbage collection um specification that we have uh that's part of it. That's part of the reason why it is is because you can just push a blob up and and walk away and never make another request.

A

So there's a lot of things that we have to accommodate that that's very possible and you know clients don't communicate intent. They just say I'm putting a blob, I'm getting a blob, I'm heading this manifest. um So we really don't know what a client's doing request for request. You know request for a request um yeah. So let's look at the um we're going to get the manifest kit and then we're going to look at the blob get.

A

So those are pretty simple and let me bring up my thing.

A

Yeah, so you should see nothing um so we're gonna do handler's manifest yeah, so we're just going to go right to the handler the app the the part where it routes this is is pretty interesting. So one of the things is like the head and the get method is, I think, always the same.

A

It it works out. It works out more or less okay. So now everything uh this this, this method is repository scope. So everything is about um a particular repository, so you'll see um everything's kind of scoped under under that in a way that, like you know, when we get a blob, that blob doesn't isn't just in the registry that blob is associated and known to that particular repository.

A

um So let's see so we we have um this thing. It's a little constructor, that's fairly new. um We have the split in the registry that you know some requests are done by the database and some requests are done by the file system and, as you roll out to production, to do that.

A

The same instance of the app is going to take one request, and maybe it goes to the database, take another request, and maybe it goes to the file system. So there is um there's really two registries overlaid on top of each other and that's going to make this dive a little more annoying.

A

So I think I'm going to focus on the file system, one because that's what that's, what you're working on right now uh jaime and it's a bit it's easier in some ways in some ways, it's more complicated.

A

So um if we go down here so this is just we get. We figure out which one we're supposed to have right and we can just not ignore it in the body of the handler here.

A

So it's either going to be like a tag, as I said, or it's going to be um a digest. So that's these lines here.

A

My tag get my digest, I would say 99.99 percent of the time getting it by tag. um I think some clients will also try to like hit it by digest to make sure everything's cool, but um tag is what people usually do or what clients usually do. Rather,.

A

So yeah, let's, let's gloss over the fact that we got a manifest a digest and an error. So the digest is what you'd expect it's the the it's, the content, addressable um digest! This digest the content. So that's how this is stored on the file system. So it's pretty important and we also have the manifest- and in this case it is a distribution manifest type. So I'll go over that real, quick and there's not too too much with this type.

A

So we can return the references which are every everything that the manifest references. So if you have a manifest list, this is a list of manifests usually and if you have a uh a regular manifest you're gonna have the config first and then all the layers, you're gonna, get from these references and the payload. It will turn the media type, so the media type for oci manifest media type for schumer to manifest the payload and an error. The payload is the the bytes that are actually like the manifest json as you'd see in.

A

Like an example of, like you know, here's what the manifest looks all built out. So, there's not a lot um really. You can do with that. um So what you're gonna see a lot in the code is um there's there's a split. um This is a fairly new interface, which accounts for both oci manifest and shima 2 manifest. So it's everything that manifest is plus you can get the version, which is just the shima version, which is just an integer and then the um I'll just oh, my gosh.

A

uh It says the shimmer version and it's the media type of the manifest. So it's not it's not too complicated. It's just. You know what what kind of thing is this? um And in particular this one lets? You get the config and the layer separately rather than all at once with the references and also lets you know, the total size of you know the manifest like the manifest content itself, the layer and config blobs.

A

um So that's that comes up for the database. It doesn't come up for the file system, so you'll see a mix between this interface. This interface and type switches- because sometimes you, the concrete type is, is important and in a lot of the older code, the types which was just how you would diff differentiate between shima 1, shima, 2, shima 1. We pretty much don't use anymore, there's still some in the code, but the api can't handle it anymore, um which is great. It made the code a lot better.

A

um So how you differentiate between the different manifest types used to be a lot more type switching, and we still have some of that. But it's reduced a little bit so.

A

Please excuse me while I flip okay, so let's just say we got it by tag. We have the manifest. We have the digest for. Let's pretend there's no error, because.

A

Let's pretend there's no error so now what we need to do is we need to determine the type of the manifest, um and this is the older code. So so we do have the schema one here. We still need to check this out because there's nothing.

A

You know older older, um you could hook a new registry client to older object, storage, an older file system and have encounter manifest human ones. So you want to be able to say um this is a shimawan manifest.

A

We don't do that anymore right. We don't want to just panic if we get if we get these kind of things or an error, that's just like I've never heard of this before, because that's not really true, you know.

A

So we also have shima, 2 and manifest list, and if it's a manifest list, it could be um an oci image index and if it's a like a regular manifest, it could be shima 2 or it could be an oci manifest.

A

That's just a bunch of um checking there and then this here is like hey. It's manifest even one blanket. We don't support it like end of discussion um and if we do shima, if it's ashima, osi shima, we check if we support it. If it's- uh and this is like um this is the client- we won't go too far into this, but the client needs to communicate like I know how to handle an oci manifest. I know how to handle uh an osi image oci image index.

A

It used to also need to say, um and it will it will send the information up still in the same way, but I used to also need to say: hey, I I support shima 2 manifests, but we dropped support for shame, 1 and now we just assume that any client is going to support. Shima 2 manifests um by default um and yeah.

B

A

Yeah, so if we have a shima 2, if we have a manifest list, a docker style manifest list, so that's um not an oci image index.

A

If they don't support that we're gonna we're gonna, try and they're getting it by tag since, if it's by digest, we can't um we have to return something that matches the digest period, but if they're getting it by tag, they don't support the manifest list, we're going to give them the.

A

If we can find it we're going to give them the amd 64 linux manifest from that and we're just going to give it to them, and it's just a fallback for for clients that are not supporting manifest lists which are technically still experimental, although they're pretty widely used at this point, so you'll see things where I think we saw some issues where it's like: hey manifest lists, aren't displaying properly and it's like. Well, that's because the client isn't saying I know about manifest lists, so you're just going to get the the one image.

A

As is you know, this is probably okay. You know it's probably linux, 64.

A

um yeah so and then we get the payload. So we get the the media type, we get the payload, we set the content type to the media, type content link to the length of the payload.

A

We you know set all the headers here set. The e tag write write the um payload to the body and um we do some little logging here, because we we keep a lot of metrics this way, um we keep some metrics this way. So yeah, that's the that's the get. It's it's relatively straightforward. The way it gets tricky is um what's the support looking like do we need to rewrite this?

A

That's that's where the extra logic is besides just getting it from whatever storage we're using. um So if all that's kind of clear or makes sense, uh I'd like to move into like actually getting the manifest from storage. If that's all right.

B

Yeah, that's it's straightforward and I'm sure missing. You know details and maybe a bit of the background, but it's so far so good likely.

A

So, let's just do get my tag um with these, so that's the database we're just ignoring the database because it's because we just we just are because it's easier. If we do it, so what have we got so the the file system? Manifester?

A

We have the tag store, which is here. We have the manifest service and then we have the request.

A

So that's all the request, information um and when you see these like manifest service, so it's repository manifest service, it's repository tags, so these are by default, they're scoped to only the repository. Only the you know the part of the request.

A

That is not the reference it's just like this is where you say alpine or this is where you'd say: nginx um our get lab org get like our container registry right, um so that that scopes, the tag to I'm only thinking about tags in this repository, I'm only thinking about manifest that this repository knows about um so, let's go through, and so what we'll do to get the tag? Is we get the tag from the tag service?

A

Great, so we pass it the name and what this does is it finds the reference um for the tag, so it gets the description, the descriptor for the manifest, and what is the descriptor well, a descriptor is uh describes, describes things that are you that are um contradictions words. This is a manifest, can have a descriptor, a configuration has a descriptor layers have descriptors. These are just the most. These are very general.

A

um Oh man can't have scriptures. Can I doesn't matter no we'll get to it later, so um the scriptures are just very general, so media type size the digest very rarely urls, sometimes annotations platforms if they are manifest list, manifests to say like hey. This is this is for linux. You know this is, for you know linux and d64 right, that's the platform, so descriptor is is a really basic thing that says like this is the digest. This is the size. This is the media type, it's very, it's very simple.

A

um So we get the descriptor and then what we see first is like hey. Does this descriptor? um You know the string of the digest. Does it match the e tag in the request?

A

So if a client says oh give me give me the latest right and they already have you know the manifest that's the latest on their system, but maybe they pull it as like v1.1.0 this checks and says like oh well, hey, that's actually, you've actually got that it's this and we can just say: hey the etag matches like you. Don't need to do anything.

A

You've got it so very nice thing here, and so then, if that, if we attack doesn't matches, we really don't have it then we're going to get by digest, which is you know, e-tag match.

A

You know again because sometimes they'll, if you get by digest and you send the same digest, you need to do an e-tag match still, and this is very simple: we just go to the manifestor get so there's a lot of these are very abstract. These are. These are mostly just to smush together the database and the file system metadata.

A

You know before this is all in the handler method like they get the handler callback, and it's just not. um You know once you overlay the metadata for the database and the file system. It helps to just have something that splits these out, so we're going to dive into the actual source. So let's look at tag store, git.

A

um Actually, let's back up even more like how do we get this? How do we get the tag store? So we get this from repository tags, so registry storage tags.

A

So we're just going to instantiate a tag store, it has the repository and it has like a like the blob store, so um not not a whole lot there, it's easier to see what you're in the tag square, so the actual tag store, will jump to registry storage tag store and let's look at get here.

A

So all the file system metadata. It's all all just there's paths right if you think of um let's go to pass so you'll see the layout like this, you know so: hey you've got the repositories manifest tags layers, all this kind of stuff.

A

You know it cr, there's a there's a link, so we're going to refer back to this when we're talking about the tag or the the file system mediator, because it all goes to this uh tag, spec that the drivers actually get from the back end. So um I believe is this. So this will be. You know the root, b2 repositories, let's say: nginx manifest tags, the name of the tag current link and that file will uh contain the digest of the manifest.

A

So let's look at the tag store again.

A

So path for manifest tag, current passback, so that's what you get here. Is you get that um sort of template field out so and then we read link. So that's just we just get that digest out more or less right and hey. If that path doesn't exist, hey that's unknown tag and then we return the descriptor. You know, just like you saw it's pretty simple, we're just opening up the tag and saying like. Oh it's actually the points of this digest.

A

Anything unusual so far.

B

No, I think it it makes sense yeah. So when I, when I start playing with it, that's.

A

Yeah I mean it's always this. This will sort of put things. Hopefully it puts like hooks in your mind to be like. Oh there's,.

B

Something there it is because I was I was looking at the content in my local file system. I could see that directory structure. I could see the you know the json blocks in the manifest, so it kind of makes sense, and now I get why we have a current and an index and some some of the things.

A

Yeah, so let's look at the manifest store, so we're gonna get it and we get it by the digest always on the file system to get a manifest. You need to digest. You can't there's no direct, look up by tag, you know. So if you have a tag, there's always two requests to the back storage backend, because first you need to get the digest from the tag and then- and only then can you actually um have something that the manifest store knows about so.

A

Let's say so: the first thing we do is we get it by the blob store, so this this thing the blob store. This is this is not any blob store.

A

This is there's there's two blob stores, so that is, I don't know if you see this at the bottom, that is a linked blob store and there's also just a blog store, but they just call them blot store a lot in the code when they're, instantiated and embedded into another struct. So.

A

How on earth do you get the? How on earth does this work, uh so we gotta get a lynx blob store.

A

First, we gotta take a detour so.

A

There's another http request, because we've got a stat um we're actually going to stat the um the repository's ability to access this and I'm gonna. I'm gonna- I'm gonna gloss over this because in the interest of time, but so we actually have to make a stat request to say like oh, this is actually accessible by the repository um and the stats a little bit weird because, like what stats it's the.

A

Blob access controller, which it that gets them back it starts to get. um Can that that's part of like the caching for blob descriptors, which we have turned off in production now. um So let's go down to the the stat method that fall back on, because it's always going to do this. So it's going to look at the blob link path and read the read the link.

A

So there's like two layers of interaction right, so you get the tag and you look up and say like: oh: what's the it's, this digest and then use that digest, and you do a similar thing um to check and make sure that that digest is, is accessible and all that kind of stuff. So.

A

Look at the link path um and that's gotten from the link path function, so the block the link blobs. Oh my gosh, I don't know why I just moved away, but the link bob store. I'm gonna cough. Excuse me.

A

A

So the link blob store has two sort of um double duty, any anything. That's like any layer and direction it sort of handles like it handles like.

A

Given this digest, can the repository access it and it works for different kinds of like a manifest or a blob they're stored in different places, but the link blob store just takes that link, function and says like um and like it just it just figures out how to get it that way. So it's the same code, but the thing that changes is like the path that gets for a digest and we'll look at that now. It's the manifest revision link path.

A

Yeah, so that's.

A

Here yeah, so this is the repository scope manifest. So it's v2, you know, repositories name of the repository, manifest revision, the algorithm so 256 sha-256, almost exclusively, then the digest and then a file called link and that link contains the digest.

A

And basically this is here for.

A

If a digest is shot with a different shaw hash, but I have not seen that ever, but it's possible, um that's why the algorithm is variable, but that's one of the reasons. The real. The reason that, where we see in real life, is that hey canvas repository doesn't know about this manifest.

A

That's what this does we're just.

A

Okay, so bob store get so we do next, let's check that way, and then we actually get the content from the other one. The blob stores blob store, which is the this one, is.

A

It deals with the other side of the registry, so if I may go back to the past thing again and sorry, I'm jumping around this is just following the flow, and it does jump around like this. So there's there's basically two sides of the registry uh like files and storage. So there is this. This is highlighted.

A

This is the repository scoped metadata. Essentially, so nothing is really stored here, long term, the own, with the exceptions of uploads um that'll, actually have content, but only for a bit.

A

You know only while it's being uploaded- um and this is the other half the registry. This is where all the content lives and is completely uh deduplicated. It is, it is instance wide it has no metadata. It is just content, addressable storage.

A

So that's where the the non-linked blob store interacts with this and it's quite a bit simpler.

A

Story um so when we do get, we get the path, and then we just call the driver directly to get the content out of the path you know it is very.

A

It is quite a bit simpler and this mostly- and this mostly figures out the path and passes it off to uh the driver, just coordinates the driver, and so the raw storage driver, which is so we're gonna, get like get content from the you know, get content yeah. It just means it just like uses the driver.

A

Basically, um so it's not that it's not! That complex, like the blob store, is just going to reach into the actual content, storage and get stuff out of it.

A

So now that we've got the content, um that's great, um unless we don't so that's just like hey there's, we couldn't find it and that's either that it's not in the storage back end or the link. Isn't there in that repository what we do encounter on production, something is really interesting is that we will get zero length content. That's you know it shouldn't, be zero link, there's a canonical, um there's a canonical zero length, blob digest but manifests have to reference something they have to have some kind of content to be valid at all.

A

So for some reason we do encounter this occasionally and we would just return the special error here. It says: hey like this is an empty manifest and it's it's gonna come up when when we do um importing into uh the the metadata database once we start doing that in production like we need to watch out for this and make sure like hey like there's nothing, we can do like. Don't stop.

A

You know don't stop on this error, it's just something that we can't really remediate, but once you have the content and you really really have the content, um you uh unmarshal it to get the manifest version, and so this will just get that uh the shima version and the media type. And so if the streaming version is one we did human one handler, which is just which we can. We can simplify this code now we just haven't. This is just going to go to something that says like you can't get this, it's it's uh well!

A

Actually I think we do on marshall or maybe anyway. So that's interesting. We don't do this in the api anymore, but if it's shima 2, we do this type switch basically, but except with the media type manifest so like hey. If it's machinery manifest like here on marshall, like his ashimaju, manifest same for us so uh same for uh oci, all this kind of stuff, like figure out what what type it is, unmarshal it for that type.

A

And that's it you know so that so there's like a there's a little bit of a validation step for the get, but only only enough to for access uh like make sure the repository is scoped right and only enough to turn it into a manifest. That's just formed correctly.

A

So yeah that's manifest, put a manifest get rather, let's manifest get and that's the simplest and it took an hour.

A

um So what we can do is.

A

Look at the blob get camera. I think I think we have time to do that. Do you have any questions.

A

Not now, okay, yeah, it's it's fine, there's! A lot of this is this is very download uh kung fu into your brain from the matrix. I bet, so it's very slowly yeah.

A

So this is a little easier because the the link blob store, the blobs, writes all involved. We've done a lot of these steps already with interacting with file system metadata and how that's laid out and how access is controlled that way and the actual uh getting the content off the common blob store.

A

um So this is going to be pretty quickly a lot quicker and you see we don't have the same sort of like um everything is in the handler. You know it's a it's. It's a lot simpler, so hey! If it's the database, like whatever.

A

We're just checking for does this exist in the repository so for the database does the bubbling exist because we've already talked about it? It's right there, it's hard, not to see to say something um which is the the stat hey. We've stated it great.

A

um If we, if it says correctly, we have the descriptor and then we serve the blob and so there's a couple things that can happen here and that is we. We directly uh take the connection and we we we serve the blob.

A

We we take multiple requests to download all of it um and that's what you get with like a file system driver, um because it's just it just lives on the registry quote unquote um what often happens with um object stores is you'll, actually just we'll just pass along a link that says here's a url where you can get the blob that you want.

A

Don't don't come back um more or less. So there's not a lot here that we haven't discussed with the uh the storage. So let's look at getting the blobs object and so because the blobs are they're repository blobs. So let's look at and go through, that sort of um that sort of situation with just the blobs.

A

So the repository blogs we'll go back to storage registry registry go and this is where all the storage stuff is like brought together and coordinated, so so restaurant. So it's a link, blob store.

A

As you can see down here- and this just like it's just the configuration for that basically- um and we do have a what's interesting here, so the blob link path is what's interesting and if we go to the paths again blob layer link path. So that's just there's a layers folder, it has the algorithm, and this is just the layers that um the repository can access.

A

um It's it's a bit of a weird. It's a bit of a weird thing: it's it's! It's it's easy to forget it's there um and there's. Definitely um like the offline. Garbage collector doesn't clean these up for one um so that they're easy to forget about in a way.

A

But there is an api endpoint that will delete this link, so you can have that um link be severed. If you don't want that content and it's it's, it's a bit of a dirty thing to do, because um now you can't pull those manifests.

A

If that's at that, um you know if your client decides that you need that layer. When you go to pull the manifest that um references that layer you it it's going to error out. So it's a bit of an odd thing, but apparently it's for security reasons to hey, like this layer shouldn't be up on the registry. It shouldn't be accessible by this repository. You can just get the layer and delete it.

A

um So it's a bit weird, but mostly it's for um access check and once the layer is there, it's usually not going to go away, um and this will come up when we push blobs because it gets into blob cross repository blob mounting, which is great and you'll love it.

A

But we don't have time for that. So.

A

Let's look back at the all right.

A

Storage industry, storage.

A

Okay, so the blobs are just a link. Blob setter, link, blob store rather, and so what we did is we saw that we stated which we just went through before we're gonna, do the same thing: we're just gonna. Look we're just gonna. Have the storage driver interact with a different path that layer link path that we talked about um and what we're going to do is serve blob, which is basically going down to the the real blob store.

A

Whoops bob store to get the surf blob block store.

A

Oh, it's blob server, that's it! It's there's! A blob. Server object! Okay, so this serves the blob and there's a lot of stats for access.

A

They're all over the file system, metadata, and so things that you think are a single operation are quite a bit of chatter back and forth to the backend. So this is basically hey.

A

Can we can we redirect this repository because it's a repository, scoped redirect turn on and off? So, um let's see so we basically just asked the the driver um for the those drivers have a url for method and um yeah. We can just say.

A

At the look at the url to download the blob easy um so sometimes hey, we don't support this method like there's no url for the for the file system driver, for example, for example, there's no url for the in-memory driver, as as you would imagine um so, then we just make a new file reader and we set up a header. So uh we can start serving this content and.

A

Yeah, I mean that's just the whole process like what's that set up, then it's it's just some. It's mostly.

A

Driver stuff at that point: it's not super interesting or not super unexpected. You can definitely look at that later when the actual driver, but it's just a file reader, offsets all that good stuff. Nothing surprising! Really!

A

um So that's how you do it.

A

Yeah, so that's that's, that's that's pull and that hits all the high points. It's.

B

So this um this pool like this vlog is gonna.

B

The client will make as many requests as there are layers in the manifest. Is that correct? No or.

A

B

Or only the ones that they don't know about.

A

Right so you're gonna make a request for the manifest you're gonna make a request for the the configuration blob and you're going to make a request for each layer.

A

Yep, so that's that's. If it didn't have anything uh clients, I don't know how smart they are, um but I imagine when you get the manifest and it's and you're like oh, I need this blog with this digest and it's like. Well, I have that locally. I don't I imagine. The client is going to take the opportunity not to make the request, but um as a registry like we don't care, we don't care.

A

Why you're doing this like um there's nothing that would stop a client from doing this backwards, except that they wouldn't know what layers to get.

B

Just give me a random one: yeah, okay, it's a bit abstract in a sense that the registry doesn't care. It's just like one blob at a time and that's it.

A

Yeah, I think the registry tries to be.

A

It tries to know as little as it can, which is great. It's a bit weird because it doesn't. You know somebody who's used, docker or someone who's like. Has this concept of what a container image is? It doesn't track to that at all.

A

So, like that, that's the most that's the thing that's sort of like you have to put your your registry goggles on and like there's, no images, there's blobs, there's manifest, there's blobs and there's manifest the manifest references, blobs and then there's just blobs, which are blogs and there's nothing, there's nothing. You have to really think about too hard, there's just blobs.

A

Who knows what they're for.

A

Yep, so I think I think this is a natural place to um cut it off the. uh I think it would probably take another two hours to do the.

A

Pull one for the blob, I mean for the push for like one one hour for the manifest one hour for the blob. Yes,.

B

I can imagine, there's a lot of validation happening in there.

A

um Yeah a little bit, but I think I think the thing with push is that if we look at it just briefly.

A

A

A

Yeah some in some ways puts.

A

I mean it's not too bad anymore, but there's a lot more checking up the the the the the range of things that can happen is just more.

A

um You know if you think it would be the inverse of of a pull and in some ways it is. But since writing is just a more expensive operation, there's just you know things like cross repository, blob mounts and there's validation, and then there's.

A

You know if a manifest or a layer is on the registry, like the assumption is that for the most part that is valid and correct, and so when there's when you're putting there's just more about, like you said, there's more validation, there's more distrust of the content. Until we decide like all right, we're gonna push this down to storage.

A

So that's, I think where it gets a bit more.

A

A bit more tricky, as well as when you do, puts there's a lot of gets requests to do the push. um I think um you know, because the clients will say like hey. Is this blob here already, because if it is we're done right?

A

So there's a lot of reads like there's a lot of read type requests that goes with a push.

A

So I think that's what causes it to be um one of the things that causes it to be more complicated and quite suppressive for that cross-repository blob mounting is complicated and blobs don't go directly to storage, they go to an upload directory and then they go to storage. So there's just a little bit of a.

A

You know there's just more that happens.

A

B

Hard, it's hard to say that and.

A

If, if we, uh you know just to I'll share the screen again, but so so this is the pull request. Diagram and this is the push request diagram, so just just for uh and not to really dive in. But with this video, but more stuff happens.

B

Apart, there's a couple of loops in there.

A

A couple loops, but not enough to make it that big, okay yeah. So that's that's good!

A

That's the that's the that's a very quick version of where things are um so I mean, I think the overview is for the file system, metadata paths.go, that that's the layout of all the stuff like and then there's really a link to blob store which operates on the repository side of the registry and then there's just blob store which operates in the blob side of the registry, and everything else is just sort of figuring out where to send the wing blob store. Basically, um so that's what that's what the file system metadata!

A

You know the database is in some ways it's simpler, because databases are more standardized you're. Just like build up the object and say you deal with it, put put it in the table yeah um and it's kind of more atomic, because we just make the we just make the database due to all the complicated stuff um yeah. But that's that's about the size of it and pools are 92 94, something percent of the traffic I mean 97.

A

Maybe uh tremendous amount of the traffic are just pulls more or less, um so it's a very read heavy thing and that's where that's where we do most of the operations you're just reading you know so that's the thing I mean, I think everything everything's covered, like with the file system metadata those two link. Those two blob stores are just tremendously: do all the things for the most part yeah yeah. So I think we're going to stop the recording and I hope all of the people following along at home have completed their worksheet.

A

They should have gotten under their chair and can send it etched onto the bow of a 40-foot yacht to my home, address.