GitLab Package Group, 28 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Container Registry Deep Dive: Blob Push (Filesystem Metadata Only)

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Hi everyone I'm haley similar and we are going to go through the request flow for blob pushes to the container registry today right and we are probably not going to get into um cross repository blob mounting unless we have time at the end, because it is um it's not the usual case and.

A

It's kind of an auxiliary thing and since it basically skips the blob upload, it's um there's not a lot to go over. Besides the fact that it can happen which we'll touch on in the normal push.

A

So I'm going to share my screen and we're going to revisit the request flow chart.

A

So um if you saw the first part of the video um you're going to get more context because we're I don't intend to re-explain things, structures within the registry objects within the registry at the same level of detail just to move this along because it is a bit more complicated.

A

um So this is the push request flow, so in general we're going to um check it's the same as every request. Do we support the v2 api we do and we're going to head each of the blobs first off, so the client is going to check if the blob is already there, and this is a repository scoped request. So that means that.

A

This isn't is the blob on the registry. This is, is the blob known to the repository and present on the registry, so there's there's definitely an end there, because it's possible that the registry knows about the blob.

A

uh The repository knows about the blob it's linked, but it's been removed from the back end, that's very possible.

A

So if the blob is already there, the registry, the client just um it knows it doesn't have to push it. Push is expensive potentially, depending on you know how big the image is all right. So another thing that can happen is that blob's not there, it's either not uh known to the repository or it's not. On the back end, some combination of the two and it'll send a four or four. We didn't find it um so we're gonna get.

A

So this is where the blob um cross represent happens. We'll come back if we need, if we have time, but let's pretend that doesn't exist and uh what we'll do is typically we'll start an upload we'll do a post to the endpoint, so v2 name of the repository blobs, uploads and um you're gonna get back from this request, an id and that's the idea of the upload.

A

So um you can delete that upload. You can say I'm I'm done with the upload. It's not going to go through for whatever reason, um and then you can also upload parts.

A

So you just patch over and over again until you run out of blob to send up to the registry, you can also get the progress of a given upload by cuid and once you're done you put to that.

A

Upload you use the uuid use the digest, um so that can either be the um the end of a multi-part upload or it can be the entire upload if the blob is relatively small um and that happens by the client's discretion.

A

So uh it isn't something that we can communicate back and say, like oh, like if your bob's that big, then you know, don't don't even try it right, do multi-part upload, so the clients do decide how to what whether to try a multi-part, upload or not all right and then at the end the client will again do a head request to the block and very likely should be there if everything goes correctly, especially if there were no errors.

A

Beyond that point and after all, that's done, uh then we upload the manifest just to put to v2 uh repository name manifest and reference. So that's either a digester tag and um there's probably a whole hour of video within that last step. Believe it or not.

A

So let's go and look at um so we went overhead already in the previous video. It's just the same. There's nothing special! uh So let us look at post, so I'm gonna do.

B

A question oh sure when, when you say a blob, it's always a one-to-one relationship to asha to a digest.

A

Right, each blob, every everything that's stored in the registry- um that's actually data, so not not like a tag, for example everything, but a tag has a shot hash.

A

Yes, there's um and blobs are only.

A

They only have their shaw, hash, there's, no other identifiers so and they're a tertiary reference to a tag.

A

At least so um you know you have a tag that references, a manifest that is a blog or you can have even a tag that references, a manifest that references, a manifested reference to the blob, so there's an indirect relationship between tags and blobs, but um typically, if you're getting a blob you're getting it by its digest.

A

B

A

So I'm now going to share a different screen.

A

We're going to go look at.

A

A

Okay, I just didn't, have it? Oh, okay! Well, that's weird! Anyway! So now we're gonna look at blob.

A

So blob upload post. So this is not the blog's handler. This is the blob upload handler so.

A

So there is a blobs handler and then there is a blobs upload handler.

A

So this is what we do and again for this video we're just gonna focus on the file system, metadata.

A

All right so this is um this- is blob mount options which were just just to blob mounts the procrastinator cross repository blob, mounting that we're just going to pretend doesn't exist for right now.

A

um So what we do is we start with uh repository blobs, and this is the link blob store, which is configured to look at the um layers folder under a repository.

A

So it's the link blob store from before, and we called this blobs create um little function here. So we're gonna see what that does so we're gonna go to like blob store and we're gonna do create yeah, so it returns a blob writer, which is something we have not talked about yet, and it's very exciting that blob rider.

A

So um this is the mount prosecutor cross repository blob mounting thing again, um so we'll just pretend that we're not doing that. So we generate a uuid, that's the uid for the upload. We generate a started app type. This is um used um to clean up stale upload data, so there's a function.

A

There's a that'll go around process that runs and it will look around the upload folders that are there because.

A

You know sometimes they just don't delete the blob uploads. If you know something goes wrong, it can just leave stale data behind so this started at um gives. That um gives that process like some kind of something to work against like so. It's not deleting in progress. Uploads.

A

All right so- and this is the path four thing this goes back to the paths- dot go so the upload data path spec. So let's see what that actually is.

A

So that is repositories name underscore uploads, the id says uploads id data, and then you have the data. So that's the blob, you have the started app and you have hash dates. So um the data and the start of that are the important things more or less.

A

So we've got that path.

A

We've got some white space here that doesn't belong, but as a very annoying anymore. So oh, it's there too. Oh, that's very tempting for me. I'm sorry to take this diversion. But what does this look like this is these are the things that bother me and you can cut that down the length of this too.

A

If you wanted to oh, there is nothing wrong with that.

A

Okay! Well, hey! Do you want to review in a mark? I may all right because that's that's egregious. I can't have that. I I understand yeah. I probably probably shouldn't that's churn, I'm sorry for this diversion viewers at home, but I'm not going to edit it out, because I don't know how.

A

um So here we write the started at file um and then we return this lbs new blob upload. So what is that? And that's just a constructor for the distribution block writer. So what we do is we get a file writer right from the source driver? So that's just you know, that's just a writer. um What is that file writer yeah? So this is driver file, writer and it just handles. Like the multi-part uploads, it handles um chunked, reads and writes so yeah and we just have all the configuration passed to here.

A

So it's not a very exciting constructor, but that's all that does so we're going to go to the blog writer again very soon, but we'll pretend we're not immediately going to jump to that. So we have the upload and uh let's say it's not an error.

A

So we have this blob upload handler upload is upload great. So we write the response. um We set headers every we return the status so that that just sets it up for um the further requests.

A

So, there's nothing nothing too exciting about that.

B

A

B

Yeah, so we we we've built that uh blob writer. That's just gonna write the response right now. How do we? How do we get it? I guess we're gonna go into that, but how do we get it on the next request.

A

Oh you'll see it's a surprise so.

B

A

That we do build it is, um you know, let's look at it for upload response, so not too exciting, not too excited right. So we built it here. So we can get the upload. You know upload size.

A

You know there's nothing too exciting. Here.

A

But we do need it just to write these um upload states for now.

A

um Yeah so uh one of the tricky things that I am going to go over this now because it will get you is that this looks like I don't know you saw, you saw the secret before uh right. Nope start yeah, so.

A

There we go so this blob upload create, writes data, it does stuff it will. um It will do things a lot more than you think it's doing it. It's not just an instructor um and I think it's kind of hard to get into what it does without going into a cross-repository blob mounting. But this is not just a constructor function, so be aware, um yeah. So, let's go back to. um If you don't have any further questions, we'll go back to the diagram.

A

I think yeah yeah. So.

B

We're gonna look at this.

A

Soon, this is a multi-card upload, because that covers everything.

A

A

We're initiated upload, so let's look at deleting and upload. This is not part of like the regular flow, but it's something that can happen. So I think, let's, let's check deleting an upload and getting the status of an upload because they're very short, um they're.

A

You know they're not too exciting. So, let's get upload status.

A

um If we don't have an upload.

B

um Still, looking at the.

A

Diagram ah yeah, thank you for that which I it works on my machine. Okay, I can see it.

B

It always works on your machine, yeah yeah.

A

So we get this blah blah blah it's like if the upload's nil, but what we haven't done. Anything like every request is different and special, so um we need to look at um how we do that. So.

A

A

How does that happen.

A

Okay, so right at the end.

A

A

Just disappear.

A

Oh shoot: how is this happening.

A

I am drawing a blank. I do not know how I do not know where it is that well, we'll find out together if I've gotten this far without knowing what's happening. It can't be too exciting.

A

So, oh that bugs me so much though. Basically we just find it um stop upload response, and this is the same thing we saw um previously repository name upload id. um What's the offset, when did you start the upload? All these, the upload url, like all these things.

A

Yeah, so all these things um you'll just get same as at the beginning, same as at the end of initiating the the.

A

The um upload so look at delete.

A

Oh we're gonna find that out and it bugs me but um yeah. So if we do find the uh if we do find an upload.

A

We do cancel on blob rider, so let's see what that is, and.

A

So what do we do? We do firewrite or cancel, so we close the we close out the filewriter and we call this remove resources function which just takes the.

A

And it takes the folder we looked at and just clears it out, because the blob upload data pass back. It's not very complicated, it's just a delete, so it just cleans up after itself.

A

So we have to do it, get blob upward, uid.

A

B

A

Do you what comes in the value.

B

Looks like some middleware setting it. Yes,.

A

I I've gotta find it I've gotta.

B

A

A blob of wood in there.

A

Okay, so happens somehow.

A

Okay, well somehow that happens.

A

I am really sorry to take this.

A

Is using your id.

A

Services implementation uses teams.

A

B

A

What happens somehow, that's your assignment find out where that no.

A

I don't want to take more time. I don't want to take more time. I'm really sorry.

B

A

B

Was looking through the code, I can't.

A

Find yeah registry is a big project. Sometimes these things happen, but this does in fact work. So we're going to look at patching the blob data um and, as you can see, if the error of the blah blah handler upload is nil, it will just have this error blob upload unknown, like we can find anything by the uuid um yeah, so content type um some white space. It doesn't need to be here um yeah, so we just um copy full payload. So this is just an append operation.

A

So the the blob writer is a writer interface, so it can just be. We could have a stream of bytes written to it um and it keeps track of its offset. So it just you know cruises over time.

A

And then we can look on this. This is a helper. It's also used with other things as well, but fundamentally we just we just do an I o copy. So it's really not that surprising or that shocking it's a writer. We use it like a writer uh yeah. So then we love upload response same as we've been doing.

A

Same as it always is write the header status accepted um and then we're gonna do um complete after that. So we patch patch patch, we patch everything like all the data is up and then the client will send like I'm finished, like that's as big as it is.

A

So we have a digest, parameter and hey there's no digest.

A

We need one, because um we're gonna check that digest later, so to make sure that all the bytes came through okay and that you, actually you sent what you expected to send you being client, so requires the digest.

A

um We copy full path payload again- and this is because this this is how you would do a single, um a single blob upload. So if you have a very small blob, you will just do this. It will just send everything all over once, but they have to be. uh I don't know the exact cutoff for docker client, but they have to be pretty small before they'll actually get single single operation uploads and then we commit.

A

So all all of these are.

A

A lot of work is done by this blob writer, so um we really need to look at that because all these are just coordinating the work of the blob writer. So we're gonna. Look at that and it's a bit hard to.

A

It's a bit more complicated than.

A

Than other things, in a way, it's um it's kind of hard to like go to particular methods, one by one and like we've done for the link bob store, so um so yeah just has reading reading and write methods. What you'd expect these are not that exciting, um read right from close they're just implementing the interface they're. Very.

A

um This is just very basic, like low-level file. Reads offsets all that good stuff, byte arrays offsets you name it it's that kind of thing um so, and you can see that these whole state, you know, started at the file writer the driver.

A

All these kinds of all these kinds of configurations so commit is when- and this is the last step so you've done all the writing and either that's um your patches or just like one single commit is where a lot of the work gets done.

A

So first you'll commit with the file writer, which is a storage driver thing which is just out of scope because they all do it. They all will do their own thing.

A

uh So we're closing um we're closing the writer we're closing we're getting the final blob size, um we'll validate the blob and we'll spend a good portion of time going over that in a second, um then we do this move blob and that's a very simple move from the upload directory to the canonical path. So let's look at that, uh so we get the blob data passed back and that's going to be.

A

um This is the blob side of the registry blob data path. Spec is v2, blobs shot 256. Then it's going to be the first two hex bytes and uh the hex bytes of the digest. Then the digest and the data. So you under v2 blob, sha 256 you're, going to see a a a b and on on and on onward and so forth until you get to ff and that is for um object, storage drivers, they tend to like more prefixes, and that gives you a fair number of prefixes.

A

um I think with super super large deployments. We could really use two more, uh but you know two more digits to play with, but we don't have that and then the actual hex digest and then the data which is just the the data which is normally a tar file. But you will also see just json here from the manifests and from the configuration blobs.

A

So that's what that's going to.

A

Yeah, so that's the blob path that has a final resting place.

A

That's so much nicer, um so we stat and the block path pretty pretty standard, um and this is done, which is the storage driver. This is not the link bob store. This is the storage driver directly citing the path, um and we want to see that it doesn't exist.

A

um If the path already exists, we don't we exit now, and so we don't. um We don't overwrite things because writing is you know. Writing is more expensive than not writing, um but the the the the tricky part with this is, if that, if that blob data goes wonky, this process won't refresh it.

B

So, in that case, the client would need to delete it and then upload again.

A

Now um we, the registry, would have to delete it because clients can't decide um blobs in common storage. The client can't.

B

Actually, delete.

A

Them the only thing the client can do is untag images, delete tags, delete manifest and delete particular layers from the repository, but the registry um may never remove the blogs if it doesn't want to do any lifecycle management.

A

So what would probably happen here is that it'd just be a ticket and there'd be some data repair that happens manually right, um because even if we do clean up images, there's a delay.

A

You know if it's offline garbage collection, you know is that once a week you know once a month, it's it's like it's not quick enough to just let it resolve on its own. If there's really a problem here, but we haven't encountered that since I've been here so apparently it's pretty rare. So that's that's fantastic.

A

um That's good to know right.

A

Yeah, so we're going to um we're going to set a different path, we're going to stop the stat, the blob writer path, and this is the upload directory.

A

So um if we don't find this, we need to check and see if it is a um zero length block, that's possible, there's a canonical digest for that, which is this um that's what an empty blob digest looks like.

A

I don't expect you to memorize that, but you know that's like that's the joy of content, addressing storage. um So if um there's an empty blob, we'll just write empty, empty space.

A

So if it's not we'll complain with attempted to move zero link, contact with non-zero digest.

A

And then new blob uploaded and then we move. So this is just the driver again this this and that's just like take it from the blob rider path, which is the upload directory, put it into the plot path.

A

Easy as that, so we've moved the blob. Let's go back to commit.

B

Is there is there ever a situation where you say you moved it, but you didn't, for whatever reason.

A

That is, I guess, possible. It would be like an underlying bug in the storage driver so and all of our storage drivers are pretty well known, like they're, pretty well known platforms. I think.

A

Like during the movie, you get a problem during the move, you could, you could have a good client disconnect right, um but it it would need this first driver to do the move, fail and just silently fail and just return a nil.

A

um So it's possible, but that's like a much lower level error, yep, yep yeah. So what we did there was we moved to the blob into stable storage right into the well. There was no sample storage before, but we moved it into the you know where it's supposed to live long term. That's this content, addressable location!

A

Everything is good. What we do now is that we link the blob, and that is just writing- that link file. So I won't get into that because it's not too exciting, but you just that's just writing that link, file um and and I'll look at it and pass that go. So we have a.

A

A good idea, so it's just writing this file and writing the digest to the link file, so nothing nothing.

A

Oh goodness, nothing! Earth shattering there.

A

A

Yeah, so we call that remove resources again. So if the upload is deliberately canceled or it's it's a successful, it will try to um clean up after itself, because that's great right, you don't. The upload pressure is like we'll have to do some work on that in the future. But it's um it has some quirks. Shall we say, um and then we get this blob access controller set descriptor, that is for, uh if we're using the blob, descriptor cache.

A

So that's just um that's just setting it in the cache. If there is a cash, if there's not a cash, then this doesn't do anything. We set the state for the blob writer that is committed and we return the digest. The canonical digest.

A

So that's about it, um but we didn't go to the validate blob step and that's um that's a cool. That's a that's an involved one. So are there any questions so far.

B

I think I'm good.

A

So um you know, like we said earlier: um we everything that makes it to uh content. Addressable storage, we assume, is all checked out everything's good. So this is a pretty important step, um all right, so basic stuff, hey. If there's no digest what are we about? You know what are we validating against? So all this is basically is that we are hashing the content with the with the algorithm and seeing if it matches the descriptor that um the client passes up at the end.

A

A

A

Actually cares not actually present for the reader.

A

A

Size yeah so we're just getting the stats. That's just the file information, so it's possible to get a zero length manifest or zero link blob. So that's why that's there um so we're just not gonna like explode on that error, so we set the size and then, if it is bigger than zero, then.

A

So if the descriptor size is not the size, so um so we stack that file and we get the um goodness. Where does the size come from.

B

There's a file size from the step.

A

Okay, so we commit with the descriptor sorry yeah, so we get, we get a just, we get a digest and we get a um a size.

A

So if the scripture size is greater than zero, so they gave us a size, um we'll say like hey: this should be the size that we we found on disk, um or maybe it's zero or negative. So we don't assume that it cares about the size.

A

Yep it's this is, this is a not the easiest part, so um zoom digest so.

A

If the so, can I algorithm um so that's the algorithm of the the blob writer does it if, if it matches.

A

The the same will say we'll set this verified um to true. If we do a different algorithm, um we're going to have to download and rehash the uploaded content.

A

um So, if we're not also for not using resumable digests, we'll have to hash the full content or there's an error from what's else there. Oh yeah, so.

A

um There's a lot of there's some essential errors of hearing about so.

A

If it's full hash, then we have to download the blob and rehash it so.

A

So, let's see if the written size and the algorithm is the same and it's verified okay, so it's like if we were around for the entire life cycle, this upload it we can take the shortcut, but if we can't um check on file size, what we have to do is um we have to read the file completely from the backend.

A

And this is going to be from the upload path, so read the complete file, we copy it to the verifier and see if it's verified and that's just gonna yep. So it's a lot this. This is like this is really hard to get to really grasp in, like this quick survey like this, but um it's possible in a blob upload that we have to download all the data from the blob back end and and rehash it. So that's a possibility.

A

um Well that's how that works, um and then we check and see if we set this verified flag at any point. If we didn't, we just say: hey it didn't work out. Canonical diagnosis does not match provided digest um and we turn an error. Just like hey, it's not a good, not a good digest doesn't work for us and then we just you know, set some set some metadata and go through so yeah. That's, unfortunately, that's just really not the easiest to understand, and I don't want to like.

A

I don't want to dig too far into this just because.

A

B

A

Is like, where does that come from right? It's part of like open containers, go digest. You know.

A

So you can feel free to dig in that, but there's a you know: there's a library that verifies it, um so we do have a bit of an extra time. So let's try to go over across repository blob, mounting.

A

Okay, I've said right and the first time so start blob upload. So we go back to start blob upload and actually, let's go and look at the request flow and go back up to.

A

Yeah so let's say we check for the blob doesn't exist. We can mount.

A

We can hit the we can do this mount thing. Mount is the digest and from the repository so digest the blob repository that the local client knows knows about that blob.

A

So this would be like if you're, trying to upload a blob to alpine with the with the digestive, a you could say like I know, digest a is also an nginx.

A

So keep that in mind. It's it's about client knowledge, that's the important thing and that's decided with like a local data on the client, local data on the client, because people ask about that, and it's very confusing, and it's very scary, if you see mounted from super secret repository to customer repository, but it's fine.

A

So here we have like the from repo which and the mount digest so hey. If these are both set, we should create a bloodbound option, which is we parse the digest. We make sure the repository is a good good repository that is properly formatted. They didn't just put like some random characters that we don't it's not good. Never could never be a repository.

A

And um we get the we figured make sure the digest is the the right digest for that repository so and then we have um with mount from canonical. So that is just sets up this little.

A

Should you mount yep, which should you mount from this, the reference that we gave you nothing exciting, um so it returns the option. It's just just some data storage container. So.

A

We go back and.

A

Let's see all right, so we passed that option to create, and then, if the, if the error is error, blob mounted so that's essential error. It's not really a real error um ignore the database stuff. Then we go right, blob headers, just like we do all the time all like every place. So that's the end. It's.

A

You know that's the end of the function. Basically, if we do a process cross repository blob amount, so let's get into the um create, then.

A

So we revisit this.

A

And hey smell, it should melt. Yes, we should melt because it we should do it.

A

You know it was determined that we should so we'll do lbs mounts, so you can provide it with a stat already and we'll do this for the database um and that just skips this. But if we don't, we um get the um we get the repo from the source repository and we get the link blob store and we stat it to make sure it's. There has access everything's, good.

A

So and that's basically the whole thing right, that's hey! We started it. It's there. It's known to that repository. You said you thought you that the client said it's there. We checked it actually is there and it's on the storage backend.

A

So we give the stat size we give the media type which for blobs like this, is always going to be application. Octet stream. We get the digest and this is a little gross, but we return the digest, but we also return the result of linking the blob right so that links it to that um and that kind of does all the work right. It just says like it, writes that layer file that we saw again in past and just to illustrate that again again it all all it does.

A

When you cross your proselytary blob mount successfully with the file system. Is you write this path? You write this path with the algorithm in the link file. That's really the only action you take and that's all that repository needs to um know about that um know about that blog.

A

So we've mounted say it didn't. It did well upload and that's when we didn't get.

A

So this is we'll actually return an error, we're turning the air like blob, mounted air blob mounted right so and that's the sensible era. So we write the headers, we go about our business and that's about the that's about the size of it. Do you have any any questions any anything? I should go back over and treat more carefully.

A

Anything like that.

A

B

A

B

You know from the overview it makes sense.

A

B

A

Yeah: okay, go ahead.

B

Yeah, no, I mean this is in a sense. It's straightforward, walking through the code. By going into the details, that's the interesting bit yeah, I'm sure it's going to take me a bit of time to.

A

Yeah- and you know these are I'll upload this probably tomorrow, which is to uh yeah so I'll do this tomorrow, and so you can always rewatch it probably at double speed.

A

And um you can always review it, and I think the intricacies of intricacies of this took me a long time to grasp just because a lot happens here. There's a lot of.

A

um I would encourage you to look at the validation step more in depth in a way, that's just not appropriate for this video um just to sort of know when that's gonna happen and.

A

Most of the time it is, I believe you know, I think I think, for um you know one of the ways that it skips validation is like hey we're around for the entire upload of this blob, and it's like probably not right and probably weren't here for, like it probably wasn't the same handler for the entire 20 gigabyte blob. Somebody decided to upload so um and that's if you look at our monitoring and you look at the memory usage you'll see this like it spikes up and down, and that's what that is.

A

I think I hope that would be. That would be the most.

B

A

Explanation for why that happens,.

B

uh Is are these handlers like unit tested, or do we have some sort of integration or influences or whatever?

B

The answer is yes, so I I I could potentially just walk through the test and see what what are we testing in there? I guess or what? What what is it expected.

A

um Yeah, yes, you could so there are. There are.

B

A

There are, there are tests for this.

A

We use, uh let me share my screen again.

A

um I didn't get into this too much, because uh participants can now see an application.

A

um Blog rider test, so this is the blob rider test.

A

So you can look through that.

A

They're pretty high level.

A

They're, okay, but um so this is going to be handlers, api integration test.

A

These are the most um comprehensive tests we have. It is almost 7000 lines of tests and helpers which we're moving out to a special helpers file, but.

A

So we have this series of api conformance tests that we run against different environments. So we'll run this with the database or without the database and.

A

What do we do so we do without the database with the database of file system mirroring with um something we can't do now for reasons. So if we do run off the database, we'll um do it with migration enabled and uh file system mirroring disabled, we'll do immigration enabled and.

A

Without like a what's it right here, oh yeah, so we do it with mirror, mirror fs enabled and disabled um and there's there's other things that we can do here, um that's complicated.

A

We run them in random order, but these are. These are the tests that are just like, no matter how we've configured this registry because, like I said in the previous uh talk, we had, um we basically have two registries overlaid on top of each other, but we really have three registries that were played on top of each other, because we have just the file system metadata.

A

We have just the database metadata and we have um a migration right mode where we're slowly moving data over and you'll be around for all of that, so you'll get really familiar. um So these sort of say, like no matter these tests are for like no matter what this has to happen. These have to be identical.

A

A

And there's a lot of these. I hesitate to sh to like get too deep in these because, like there's pretty um they're pretty high level in a way because, like um we'll see them manifest and push it by tag, and that, like that's all done because we had to help, we have a helper for that. That just is like does all the stuff, because we have to do it so much in writing these tests.

A

um And then we.

A

We will, you know just make the http request against the um you know: write the environment, that's running in this test and we'll you know, check the response. We want to see these kind of error codes. We want to see all this kind of stuff, but they're super high level.

A

um So it's it's kind of hard to take these tests and like step through the code execution, because they're they're very.

A

They're very broad, but um they don't break very often if they're not supposed to so they're, not very fragile, and you know they've they've caught numerous things over the years, so I'm very proud of these at least um but as far as like, I can use these tests to understand what the handler is doing.

A

Not so much you can like get your expectation um so like what, if you get a humanity manifest by digest, and it's like it's not associated with the repository like you should get. You know not found status in error code manifest unknown, and so you know that, and so you can kind of like figure out how that would happen. I guess, but.

B

Yeah yeah, I guess this. These are the tests. I never want to break and if I break them like you know when I'm when I'm when I start working on things, it's the ones that should be like, I don't know, maybe the most important yeah.

A

These are very important. This is extremely important, really expanded a lot on these because um because they're so broad and they're just like when I do this, I want this. I want this back right and having rewriting the metadata subsystem completely for the database. It's completely different than the file system metadata.

A

um That was one of the things that was super valuable to me was to say you know we have.

A

We have tests that are high level enough to say, like, even though there's a completely different code path being taken for the most part, everything works out at the end. So that's the way. These reason these tests are like this.

A

So yep these are fun. These are. These are interesting. These take a these are probably pretty fun to write. By now, they used to be quite kind of a pain because we didn't have.

A

They used to be not broken out like this. There used to be like test manifest api, the end and it tested the manifest api. All of it.

A

And so it was like a thousand line tests yep, and it's like well that failed. It's like great, I'm not surprised.

A

Yeah, so let's just stop sharing if you're following along at home.

A