OCI Weekly Discussion, 21 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OCI Weekly Discussion - 2021-10-21

Description

Recording of the OCI weekly developer's call from 21 Oct 2021; agenda/notes here: https://hackmd.io/El8Dd2xrTlCaCG59ns5cwg?view#October-21-2021

A

Five minutes all done ready. Kick this one off.

B

Yeah, it's close enough to five minutes. Yeah um yeah, take it away, brandon.

A

Yeah, hopefully these are quick, um but I just want to throw two out there just get some attention on them and make sure that I'm not completely out of left field with what I was thinking here. um One was on the artifacts one. We had the pr out here asking to archive it and I was a little confused by that, and so I wasn't sure is there something I don't understand? Should we be using this at all, or is this a little premature.

C

I should go ahead and uh apologize for being. um I was doing aggressive uh grooming of oci yesterday and there was a number of projects, so I'm in general I think, there's a discussion that probably needs to be had I'm super confused by the status of this project.

C

I thought that there was a disagreement about it and that's why um work is being done in the aurus one, uh but maybe steve could you know explain it the way that he sees it, but I you know I didn't mean to cause any issues with this or confusion, but I, like I put here like I'm, also very, very confused. What's going on so.

D

That's cool! It's fair! It's for the.

A

D

Not going to weigh the the we just need, we got a little bit of a pause, and you know in finishing it up until we've decided if we need to manifest or not, and we've got work groups that are going getting together to make that decision.

E

Is there a place that that state could be surfaced, for I mean for josh, but also for for the next josh that shows up and says? Oh artifacts,.

D

The readme is a good place. Let's just change this pr to a you know a link to some efforts that are underway. The work groups, as well as the uh the auras effort, I think, would be fair to link to uh yeah and also a couple other places, the the potential prs, um the reference one that was done in image spec as well as um linking probably to uh mr, mr, uh you know, justin's a pull request. I think it was 29 or something or maybe uh another number in in this. In this repo.

E

Yeah I mean, I think in general,.

D

E

When a person is confused, it is the documentation's fault right. uh So if, if josh, who is even a person, who's relatively plugged into the effort and knows you know knows us and talks to us sometimes if he's confused, then like some rando off the street is gonna, be extra confused, so more documentation would be great.

E

F

I guess I support documentation, it's helpful, I sometimes it can be more confusing, so I just rather than spawn this out to several different things, the like, if you look at what's in this repo now, it's strictly here's the guidance on how to use the existing manifest that is in production. People can use today like it's, you know as released as anything could be, because that was the whole debate. Is this a spec or is it an uh reference implementation, not reference?

F

A discussion on how you can use a prescriptive architecture for how you use the existing manifest to meet a goal, the additional? What the only reason that 29 and what now is the norah's artifacts is not in here was we didn't, have an agreement that this could be promoted to a spec.

F

That's the working group discussion. um The outcome of that working group is tbd right. It could be. It could be specs that go into this one. It could be a merger with the distribution spec like we don't know yet. So I I I'm, I don't know if we should be adding more links to things that aren't closed as opposed to it needs a status section in the doc in.

C

The reading yeah, I think, just at the beginning, like jason saying, like just hey like actually the progress is going on over here. Maybe it'll come back into this like. Is that the idea that you'll work without being uh pushed back upon in the auras one and then get it back into here as a stable.

F

Thing like is that what the working group here is the question like we don't know if it should go in like when the artifacts repo was created, it was because we couldn't get agreement to make changes to distribution or image. So that was the concession I don't know what the output of the working group is like. I have, you know an opinion, but that's the point of.

E

The working group, so I think that context is useful to share with people who find this repo right like like this exists, because we tried to do this. We are the current state of it. Is it's being discussed here and here by these groups, like you know, see this issue for the latest.

E

For the latest discussion I think, uh uh uh like I said, josh josh is a person who is relatively plugged into this stuff and if he doesn't know what the latest status is, then then some person who saw a presentation about artifacts and wants to get involved will have no no help like. Where did they do it? Do they do it in this repo? Do they do it in rs artifacts? I just I'm just trying to give people breadcrumbs to find where the current discussion is happening.

E

That's all because it's it's currently happening in issues and pr's and in three or four different repos.

F

I mean I'm happy to do that. I you know we're having a big changes to this. I was trying to not add more complexity to it. Quite honestly of you know, is this a fork? It's not a fork, not that it moved. You know this is on the, maybe that's the words we can put there. If the group is okay with that, that's great, considering all of the contention that's happened around this I was trying to.

F

I was kind of surprised and not looking to create more bike shedding on various debates and, let's just let the working group evolve and go from there. Well.

E

If you're not interested in bike shedding, I think you might be in the wrong group, uh but but I think, like uh the discussion of specific wording by all means, a pr is perfect for that. uh But I think I think I agree with josh whether he meant to say this or not, that the current state is confusing and some text somewhere saying what the current state is. Even if it is like go look over here for the latest status. uh That would be fine.

C

And that- and that was really the purpose of of the pr is just like- I thought the updated thing was there, um and so it was simply when people landed this because they're coming from someone's blog post or twitter that they would then be redirected to the aurus one. Where the you know um anyway, I'm sorry didn't I should have reached out to you, know mike and steve and got more clarification. I was just like.

B

No, it they're, like, like jason, said if, if any, if there's confusion amongst us with the status of or like where, where that discussion is or otherwise, then you can expect others to be more confused. So.

F

Okay, can I just suggest we close this one because we're not trying to archive it and we'll open up another one that that has, I think.

E

F

Well, there was a couple I literally just mike mentioned something to me the other day about this, and it didn't click that it was actually a thing like in the repos. So if we can close this one and then we'll open up a pr, we can.

D

Yeah, I can go ahead or rename it whatever. It's all good, either way I'll.

C

Put a I'll put a nice comment in here uh right now. That's like this is. We need some more direction for people I'll, reopen it when we're clear on that, and I think we don't actually even have a link for a working group. Yet right that's soon we can. We can link to.

D

The the ones that are in progress.

D

Update to the readme to indicate that okay.

C

A

All right, all right, sorry, okay and and we've moved from master.

D

Domain, so thanks for the reminder on that.

A

Well, that's a good one. Hopefully I didn't open up too big of a can of worms on that one I just wanted to. I was confused, so I wanted to clarify that one no.

F

It was good to see like once I saw it on the agenda. I was like wait. What.

C

There's there's blood on.

F

C

Brandon there it is.

A

D

Don't shoot, don't shoot.

A

I'm just raising the visibility on this one, because I think it's come up a few times of people discussing different ways that we can possibly have backward compatibility with artifacts today, you know differentiating between what's going on with the artifact spec work over and over us with.

A

Potentially, if we need to support older registries, I think we keep bringing up over and over again that would be nice if there was a way that we could say, update this thing in a registry, but only if it's what I thought it was so we can avoid potential race conditions, so.

D

Exactly and that's the kind of guidance that this this repo is supposed to happen so.

E

We need to fix that yeah.

D

I think I think it's.

E

Especially useful to frame it not in terms of any specific type of thing, being racially created or updated, but like regular images, have this bug regular indexes? Have this bug right, like yeah, I think I think we can easily fall down a rabbit hole if we talk about specific use cases where this is more more than likely to happen, but yeah, regular, regular images from 2017 have this problem and registry should do better at solving it. So plus one plus two even I'll, give you plus two sargon.

E

G

E

G

You hear me, I hate zoom, um how do we implement this on s3.

E

I'm not sure I'm.

A

Not the original author on this one, um I don't know.

B

But it's mostly just in respecting various http.

G

Pieces right, but s3 has no has like correct none of that. It's like a key value.

B

A lot of this is, I mean because in the past, when we've talked about doing like static registries, where you could like just save a static tree and be able to fetch it.

B

These kind of these kind of, like basically requiring some kind of server interaction. Things is what was always the tip for tat on whether or not we should support a fully static registry like an s3 or otherwise.

E

When you say aesthetic, do you mean, uh do you mean a registry that does not accept pushes.

B

E

In that case, then, etags don't matter right, because you can't you can't write to have a read, modify, write problem. You can't write so problem solved.

D

B

E

B

Everyone we solved the problem, uh but but I think you're pushing the s3 you'd have to have an intermediary anyways.

A

Yeah yeah, I'm I'm just used to having a front on top of s3 and not talking to s3 directly as a back-end registry, so yeah.

G

We all do it's called distribution, uh but like distribution.

A

Is going to be.

G

A

So this would be distribution supporting and distribution would do.

B

Track of those keep track of the e tags that it's referenced in any 302's or otherwise,.

G

Well, yeah, I mean the.

A

E tag is the digest.

G

Right exactly the e-tax can be derived it's really doing. The cash operation.

G

Compare and swap.

E

Not content or just storage right, yes,.

G

John's cars, not your guests.

B

So so what would you be looking for? But I guess it's something if, if, in the push use case, you already have something on top of that? What are you looking for from s3 then or like how.

G

Basically, I'm saying that this spec is impossible to implement on top of s3 and as far as I know, the sd back end is the most popular back end for distribution, um and I would really like so just because you'd have just.

B

Because you'd have more than one front end fronting the same s3 bucket.

G

If you have more than one distribution node, uh you can't do a read, modifier right operation, that's consistent!.

A

So would that be a case that you would just not set this header.

G

Yeah but like let's say that you try, um I, I just think you can easily get into an awkward situation where, like clients are not necessarily going to take a negotiation uh to see if this header exists and like this is a lot of complexity.

G

uh Alternatively, I would love to see someone actually get distribution to implement this using like there are clever ways by using side channels and registry or distribution has historically supported uh caches on the side. um In other databases that use ec I've seen people build consistency, systems that are external to them. I would kind of love to see someone prototype a metadata store on top of or next to distribution, just to see how complicated it would be to implements on top of s3.

G

E

I think I think this proposal would actually help motivate that work in distribution right like like we're not if this, if this gets merged, it's not saying the distribution is bad. It's saying here is a recommendation, an oci recommendation that that distribution does not currently satisfy it's just a recommendation uh and because it's an unsatisfied recommendation now somebody can say. Oh, I would like to satisfy that recommendation and go. Do that, however distribution. I.

G

Don't know anything about how hard that is, I don't know.

E

If that's mentally easy medium hard.

G

But right right, so I'm suggesting that, like maybe we prototype it first or maybe someone prototypes it first, just to see like if it turns out to be an insanely large amount of work.

G

Maybe we should think about.

E

Some different ways to do this.

E

I think it is supported by a number of registries- maybe maybe it is mentioned already in the in the issue, but is this behavior already supported by a number of registries in the wild.

E

I know that etags are not like new technology, so I wouldn't be surprised if a registry did, but I also don't know if they all do or if none do.

E

I take your point, though, that it's worth it's worth um thinking about how hard this is for a registry to implement, because, like the.

G

E

G

See this playing out is that, like our developers are gonna love this and our users are gonna, love this and then as registry developers and operators we're just going to be like. Oh no. What do we do now.

E

Well, I don't think these things exist for the benefit of registry operators. I think they exist for the benefit of users right.

G

Right, which is why I'm suggesting that someone spend a little bit of time, thinking about how something like distribution might implement this.

E

Yeah is there anybody uh either here or uh who knows someone in distribution land that would be willing to scope out not like necessarily build it, but like give an estimate of easy medium hard uh on this I mean you, you seem to have some knowledge of how uh distribution ns3 works, so you are already infinitely better than me, because I have no idea of either but.

E

I don't know at this: it's the same chicken and egg problem of like if the spec doesn't have some trajectory sort towards suggesting it or recommending it or requiring it, then nobody's going to do the work and if nobody does the work, if nobody doing the work is a blocker on it being proposed in the spec, then we're checking an egg right.

G

I mean yeah we've talked about like I feel like this has come up a number of times in this group, where it's like, hey, propose a spec. What were the implementations? The specs look like, but the implementation takes a long time. If there's no chance, spec will get merged. Let's not do it in this case. I don't actually think that it's a a ton of work. I think someone just needs to propose what the semantics would be so like a lease-based semantic would be something that would be very um realistic. I guess.

E

The the benefit of this of etags and this proposal is that they are not prescriptive about how the e-tags work right, like like distribution, could solve it completely differently than you know. My registry or brandon's registry, or whatever like just saying etag, should be used. I think, is not a is not a an onerous thing for any registry. I get that it's worth like. Looking into how hard it would be practically inside distribution, but I don't know as soon as I saw this I was like.

E

I can't believe the spec didn't already say you should use etags.

G

I mean ignoring e-tags, I don't think that's the problem. I think the problem is the back-end store. Doesn't support consistent operations, so you need to know the mechanism on top of the back-end store to support them.

E

A back-end store of one registry implementation.

G

Azure azure, I know.

E

G

What are there.

A

This is, this is a problem if you have multiple multiple instance: distribution back by the same s3. If you've only got one instance of distribution sitting out there, you're only talking about one thing: you're not gonna, have multiple writers. In that case, then, you wouldn't have that.

G

If you have one server but like if you want, ha you need two zero. Yes,.

A

Yeah, so that that's where the challenge is going to be implementing it, um so that's kind of where I phrase it of what are the next steps for this, and so, if we're saying from those the eyesight that we'd like to see somebody from distribution take a shot implementing that and that's what we're held up on then that gives us some direction.

A

This should come as a proposal.

H

Because- and I think reference implementation needs to show that this is already supported in some place, we've asked that of every request. That's come through right, so why would we not kind of like make this as a proposal for 1.1 of distribution spec and taking that away.

G

Like I know that uh quay supports this, but qui also requires a uh sql database right and like that, has some complications around availability. um So it would be nice to be able to have a story around. How could are? Are there any registries that implement this without sacrificing availability, a ton.

A

Is that availability on only rights or is that availability also on reads?

A

G

In the k case,.

A

All right, so I think I've got the two answers I was looking at for that one.

A

I think that takes it to sargon with a new release.

G

Yes, I have no idea how this works and I could not find any documentation about like what the process is and so on.

B

Good question: uh it was the it's it's effectively. It's been the same process as the other specs that are did have some documentation in a few different places. It might not be actually all insane. If you looked at runtime spec, it would probably have it and then distribution. Spec was the first um release that somebody besides myself, actually went through the whole spec release process.

B

So josh did that for the 1.0 um and if there's any improvements, otherwise, like just copying or carrying over the release process to like a release.nd in the distribution spec, we need to do that.

G

Is there a concept of a pre-release yeah we've we've.

I

Done well, we did rc's before the 1.0. Yes, you've seen those yeah.

B

And even with the other ones, having a you know, 1.01 or whatever that followed up, we did not really do an rc we could have, but we just didn't.

B

um I imagine if we did anything like a 1.1 we'd, probably go to the rc cycles. I'm not opposed to having something like a pre-release or rc for even patch release.

G

Is there any preferred cadence for these releases.

A

Go with the open source solution of when it's ready.

G

Is it ever ready.

G

Okay, that answers my questions.

B

Okay, sorry, I have some noise going on on you, so I just pasted in the notes, under distribution, spec release a link to the runtime spec.

B

Releases.Nbc, the big things are besides, just like the tedious cleanup nonsense that we did it's just like sending.

B

um Typically, we would actually create a milestone for the things that we were tracking that we wanted to get into that release. So, if we're thinking about a 1.01- and we should we can go ahead and create a milestone for that- that's low barrier, look at things on that github milestone.

B

Then once we've done it, then it's basically just sending out an email to the list, um well sending an email out effectively trying to get all the maintainers cc on it, but also cc the list, and it has a timed voting period and then it's merged in so there's typically a little bit of a song and dance of putting up a pr so that people can see what would be tagged as oh, you know, dot next and then having a vote that the maintainers um lgtm on the list and then it obviously then also requires a couple of zms on the er and it merges and the pr that's in in that is usually like one that bumps the actual version in some file and then then bumps it back into like dev.

B

So it's always dead and trunk. um But still in that pr is the one. That's tagged actually like get signed. Tagged and signed is the release.

B

Sorry, okay! So.

F

Yeah I mean I watched the video from the meeting that was out a week on vacation on the extension api, and you know this kind of came into the conversation of. Can we get something in not the merge spec like there's not release specs, what's in maine is like the work in progress, and what's there is hey? This is what we're working on, but until a release is cut, it might change like until we have enough confidence that this will work, you know, will it work across?

F

You know is that for that example one, but it could be any one of these topics for the thing that sargon promoter proposed a couple of weeks ago um I mean we, it seems like we need some way to have something: that's a work in flight, here's the current status it may change and until a release is cut, it's not finalized, but at least it represents some amount of confidence that this is where we're currently at.

B

So, or you say, just like nuance is always saying that we'd work in like a dove branch and the default branch is like whatever the most recent release is or.

F

I mean, I guess I guess that's the question like we went through this with the notary project. We were working in a prototype branch, for instance, and that just confused the hell out of people, so the feedback we got was main is the work in progress and then there's releases for what's done so, for instance, sargon's. um What was the thing? The d-duping?

F

I think that got merged into maine, it's not released, yet it's a place for people to reference. Do the implementation, I think we're. I think, we're pretty confident about how that's in that it probably won't change, but you know, as others try to implement it, maybe it will as the referrers api another referrer. Sorry, the extension api goes in to enable things like the referrals, api or other extensions that others might want to do until they start working on it.

F

They can point at something which, in theory is a name is no way to know if it's done so maybe maine needs a header that says by the way. This main is a work in progress. If you want the stable things, here's release one here's release x and you put pointers to it.

G

I think just furthering on that uh there was a conformance test that was added and because our conformance tests are tied to our releases and the conformance test was testing pre-existing behavior, it kind of adds a s yet another layer of complexity uh to this discussion.

G

So if we want to say cut a you know, 101 with one of the conformance tests.

G

Do we do that? Do we say that this conformance test is going to make it to the next release? And you know the associated features are likely to make it like how to kind of track this, and you know I would love to see that conformance test start to becoming tested against registries in the wild, because I know they're non-conforming registries.

G

um So there's there's that complexity too.

A

That would follow simver. If we said a 101 is just a bug fixed on the 100 release, and so people should be. If you're saying you want to conform it, you should be 101 conforming.

E

Is that documented somewhere? I agree with you, but is that something a rando off the street would understand by looking at our documentation.

A

I feel like I'm a randall off the street, but yeah it probably shouldn't be documented.

E

You're! So much more than that to me brandon.

E

I think the the 101 question does that then also apply to 1.1 because yeah that's.

B

What I'm sitting here thinking like because one of the things that it came up in the past- and we said, let's just to get tags for now like main and then tags and not actually get them to get branching because hey we didn't want to like maintaining branches and merging back and whatnot, but for some of those performance tests to be able to say like whatever this. This moving target is is always b1 and then it could be v1.1 or whatever.

B

And then you have like increased amount of cherry picking and merging, but um to have something like a moving target so that, um but for now it's just a git tag strategy.

B

um But as far as performance classes, tests being able to reference like the latest would be one at least in b1.1 or whatever december kind of stuff, and right now that would be wiring that up to point to git tags.

B

So, yes, I think people should be using the performance test, how best to have a recommendation on what.

B

uh Help us to attack attack that and make sure that it like is consistent or otherwise, and even like for, like fighting issues that we want to discuss and get it in. You know it's all in the main or whatever.

B

So um let's go ahead: let's now, I'm I'm thinking out loud, you thought it'd be more convenient.

C

No uh so sargan the conformance test binary should be released with the tag. If it's not, we should uh fix that and then there's also an image in github's registry. That should be tagged with the release. So my thoughts are that you could run the tests locked in that version and you would say I'm a 100 compliant and then, when one one comes out, it's a whole new binary. That's locked!

C

Let me make sure that's there, but yeah.

G

But if there's language in one o that there wasn't the conformance test for, should we then release a 101 and say oh yeah,.

C

I would think 101.

G

I would think 101 yeah.

C

A

And then to the question of what what happens with 1.1, I think that's when you have. The new feature doesn't necessarily have to it's, not a breaking change, just a new feature and then start adding tests on that.

G

So if I'm just to recap, our version december is uh z, increments, are clarifications and additions to the conformance test to check all previous y releases against those conformance tests. Increments of the ui release may add new features where the future version of the tests will validate previous versions may not.

A

Yeah, I always look at the last one.

B

And in between the accounting term is current and perspective, not retroactive.

B

Current perspective, you're, saying previous versions of the test might not might not be able to test for those things, but the current conformance test, and otherwise in future versions of the other testing right.

A

And I would include those bug fists because you know just typos in the spec things like that. There's all kinds of stuff people want to put out there and fix.

G

So how do we get the conformance test? Releases 101 just need to tag. We.

D

Just need to the maintainers of the repo just need to talk.

G

And and somebody needs to play, you know, get charity picking and you know if.

D

Necessary we'll take a look. I don't think anything was in a required, a 1.1 release. I'm pretty sure the change is raw, 01 type thing so we'll get to it. If you, if you want to open an issue, that's fair.

D

If you haven't already.

D

I don't know if we need to use what run c is doing, which is you know, more of a quarterly or whatnot cycle to grab up what the current one is. If you think.

B

That's a there's, there's not even a release tag on some.

D

D

So it's just an image and and a commit number.

E

All right, if that topic is done, the next one I had. Oh, I don't know if it is done, is this topic done? I don't want to cut anybody off uh the next one is mine, which was mainly just a fyi in case people didn't see the reference types uh working group proposal was updated.

E

Take a look, I don't know or if people want to discuss further, uh but I'm not sure that that was uh communicated like I didn't see in the slack. But if you are interested in that proposal, go take a look.

J

Is the reference type proposal or the sorry, the working group proposal ready for tob votes, or are we expecting for additional revisions before.

E

A good question to which I don't know the answer, or even I don't also know if that was a question to me,.

K

Yeah, so I yeah, I think, we're effectively ready for tmb vote. I think there was a call for one more participant that lackey put in that in that issue, but other than that, I think the text of it and what's being proposed is is basically it's time. You know for a to b approval.

E

Exciting, that's all I had. I just wanted to bring it to people's attention and having now brought it to your attention, we can proceed.

K

Yeah, that's a good point that if you're not following that, you might have missed it. So maybe we'll just send a quick note to the dev list so make sure people see that thanks.

E

Especially if it's ready for it for tob boats or as it becomes ready for tob boats, um yeah.

E

The next thing was content encoding. I think that was sargon. I think.

G

Yeah uh it's the next one is interesting, kubecon conversations.

E

Oh, I moved that below yours, because yours is actually technically related to things and I just wanted to.

G

Oh yeah, so I'm gonna reopen this wound. um Oh good yeah, I I don't know if people were familiar with this issue from a while ago, but there was no issue, but you know a proposal, but there was a discussion around adding content, encoding and accepting coding support to the registry uh or to the distribution. Spec excuse me, um rather than using our own um media types and media type uh uh based um uh negotiation.

G

um So I just wanted to see if the proposal was clear on what except encoding meant exactly and if it was clear, then does it make sense to go? Do the dance of building a prototype of what it might look like.

G

And just to refresh everyone's memory, the basically the only thing here is when a client does a pull of a blob. They say they accept encoding, identity, gzip, xz, bloody blah, and we then uh serve up their preferred encoding.

G

In 99 point, whatever percent of the time until people start moving off of charges, e that will be uh identity, encoding.

B

So is that is this like? Would you say, was this a transitional first step so that you could potentially stop even having compressed images across the like the thing that the identity and digest is calculated on being with any compression associated with it, so that you could on the fly, do zysta or xz or whatever across the wire, and then identities move.

G

More towards just uncompressed right and you could even do stuff so there's a second part of the proposal which I've decided to split off from the first part, but you could upload a tarball, that's not compressed with the digest. The tar ball, that's not compressed with an encoding of gzip and uh you would verify the digest on the uncompressed form, upload the compressed form. And then, when people go to download that blob, they only have the permission to download the blob with the encoding of uh gz.

G

Let's say that that would be one of one route. We could go down.

B

Well, it's neat, so um I'm all in favor for this I've I've despised the fact that compression was tangled up in our identity mob identities for years, um especially that it's inconsistent across places that prototype sounds a little tricky, because there were brief, fleeting moments that different compressions actually had vulnerabilities in the decompression piece of it so binding that up with the client to have to do that, decompression just to attest the identity of it, without just doing like a straight opaque, digest like checksum, it's a little challenging in certain.

B

It raises certain security fights, but it's a fine prototype or you know, kind of a demo, but on the whole, storing, storing and having ability based on uncompressed blob and then finding other ways to like say you know here's here. It is also if it's xz or z, stud or whatever, like ways that you could actually present it or point to it or even move it across the wire types of compression.

I

I think it's useful and neat uh needed. Are you.

G

Talking about where people are building like zip bombs, or are you talking about where people are building incredibly large documents, um because the content length can't be calculated calculated ahead time.

B

No uh there there were, it was more actually just got down that that that issue. That happened right around the time that we were arguing this and it kind of put the nail in the coffin I'd have to go actually do some archive spelunking.

B

I think it was lzma early xc had an issue where all client implementations actually had like a memory buffer like overflow, just in trying to read through and so just in trying to get to check some of the uncompressed bits you'd have to run through.

B

You know, you'd have to do processing to jump through to get a digest and it could potentially have like a buffer and as well and like particular vectors, and it was in the kind of implications it quickly and easily fixed, but it kind of like put a nail in a coffin from a conversation that needed to continue five years ago, right, okay, so then yeah, um basically anything that needs to digest should be so opaque that you can do it with command line tools.

B

Not you know like like shot once on this object, not having to do processing to get that digest. um So.

G

Cool, I can make money.

B

In the other hand,.

E

You may have answered my question, but I'm too stupid to realize it. uh I have an apology also if this has been litigated and answered before uh uh the size.

E

The size that would be returned is the uh uncompressed size right like like I get a size and a digest the http content length. Is that we're asking uh uh not exactly so. I see I see an image and it says layer, one or layer. Zero is digest, abc size, a thousand bytes, and I say oh, I can. I can process a thousand bytes. That's easy I'll go fetch that digest descriptor size. Thank you, yeah yeah. um uh Then I go request that zipped.

E

I say I yeah, I don't wanna a thousand bytes, it's actually too much for me. I'd rather uh fetch less than that and then decompress it on my size. On my side uh I am doing a lot of trusting that that inc, that size isn't going to actually be bigger or even way bigger, like the the descriptor size, is there to prevent me from to.

E

Let me put a cap on how many bytes I'm going to pull over the wire right, and if the compression used is really bad, uh it could be way more than the thousand bytes of the uncompressed size in.

G

Theory, you would get information in the http header, saying the size of the compressed size right and then the descriptor size would be the uncompressed size. So you could allocate the larger of two buffers and then decompress in place or something similar to that.

E

And so so, okay, so I'm still guarded because I could say uh I'm fine downloading a thousand bytes and then I find out from the content length header that it's actually two thousand bytes and that's too rich for my blood. I'm not gonna, try to download it anyway, right, great, okay, but.

G

In that process, I.

E

Have walked away from a transaction and made the server do the compression for me.

G

Oh so I hope that no one implements this by doing compression on the fly sure.

A

G

Way that this works is that when people do dock or push- and we did the multi-part blob assembly at that point, we would do the compression and output the eight different compressed formats that are supported. I would hate for people to actually do on the fly compression on serve time. I will for fun, but I take your point.

B

Yeah, that's that's. That was that's where we got to in the past was actually um almost like what I did with the crazy tar split project was going down the the all the different ways that diffie-hellman windows are calculated for all the different gzip implementations to see. If we could like re-read and calculate enough to re-assemble whatever gzip they used in whichever optimization, but it was golang or bsd or otherwise, um so that we could do that on the fly in the past. It was really maddening.

B

So it's possible it's academic, but it's not there's no profit in that.

B

F

One of the things we talked about at the offsite was uh usa summit was, and it just triggered my idea. The digest thing is: do we have somebody? That's gonna experiment on the 512 digest to see when that crisis happens, how bad it is across all registries.

B

What do you mean crisis just the length of the string or.

E

B

E

Shot 256 might get broken tomorrow and if it does, oci falls apart I mean everything falls apart, but one of the many things that falls apart is oci. I don't think steve to answer your question. I think uh I was the one that talked about that at the summit and I think everyone just sort of shuddered quietly to themselves and we agreed. We should do something about this, but.

B

Yeah, that's what I'm asking it'd be a massive they'd, be a massive recalculation, there's some parts of me that wish that we could do some kind of multi multi-hash, but basically there's been work done on this and prototyping and like everybody's, familiar with the fact that you can change those things and as long as you're doing it for yourself and you won't break anything, it's the fact that we trust stuff that other people have built it might not get rebuilt.

B

um It would break uh even sergeant mccall's already like pushed out a lot of work on the blake to be otherwise um as that one's gotten vetted.

B

um But yeah, are you just saying like computationally what it would look like for for people to validate it or.

F

I I guess my takeaway from the conversation was we'd spin up. You know a group that would start experimenting like here's, a conformance test, here's something that you know somebody could start running on their various registries to see like what happens, I think you're bringing up a broader thing, which is a great conversation as well. So when this does get broken, do we just accept that for existing content and don't change it? Do we fix the existing content whatever?

F

That means, because everything kind of falls apart, because all the links are based on digest or can new content be start being pushed within three days? And what is the expectation of registries that you know implement the spec? um Does that make the 256 character limit that we have on things like? I think that's on repos, I don't know if it includes the tag.

B

One one thing that it one thing that I I think in my mind all existing content stays there like addressable and everything. Should you know capital letters should migrate to something else like prop 12 or otherwise. It's not currently broken um and those things could even be effectively interchangeable that you might have like a from you know like one of the layers in the stack has been a shot 256 and the thing you just built on top that has changed since this right, you're w.

I

B

But your new image is 512 for at least a couple weeks, yeah, um but yeah, I think doing a spike like that should work, and I feel, like people have already done that. I have not, um but people even on this call probably done what would that look like with the different checks on the shop 56?

B

um How to give a warning like hey you're, still using chapter 56 could be just like guidance, for you know, different implementations.

F

I guess I was thinking, is there you know something to do with the company? That's could could I could this be done in the conformance test? Is, do you support 512, and you know just just brainstorming ideas.

E

I think targeting 512 is even a mis like targeting a single thing is wrong. If, if we're going to do the work of getting to another digest like format, we should do something like more extensible it for for uh conformance tests. I think I would accept it as a conformance test to say I support md5, which is worse, which is like the wrong direction, but at least it proved it's not as a guidance for what you should use. Instead, it's look at how flexible we are.

E

We can even support rod 13., um something even even worse. um Anyway. Brandon has his hand up. I didn't mean to interrupt yeah.

A

No, I don't think we need to rot through things that wouldn't be a bounded length, but uh going on in in general. I think I think the one question I have is what this looks like in a transition state um and maybe that's in there, and I don't think we would be able to answer that in five minutes, but just in general.

A

Something to think about is, if you have older clients doing the push, or maybe you have a newer client going to push an older client during the pull you know, because the content addressable store it's that digest has to match whatever they're. Seeing if it's an older client doesn't know about the about the encoding happening on the fly, and so those are the scenarios that I'd like to see fleshed out.

C

um So I actually, I think I want to say me and jason talked about this, um but basically one of the things with conformance tests and distribution is they're, pulling an image, spec library. So the the writing of the spec is saying. You just need some document that references the blobs, so I'm not even talking about going away from sha-256, but I was considering like just as a proof of concept.

C

Writing like a yaml based uh registry that just converts in and out of yaml, and then we could provide to the conformance test like here's, my manifest and descriptors and they're actually ammo, and you throw it against the registry and it works given that new. Because then, if you introduce a new image, spec manifest type or any of these new things like uh refers, then the test would be extensible enough to like you, just pass it a bunch of blobs and then those somehow get templated into these new things.

C

Does that? Does that make any sense.

E

Maybe not, I think, it's a different dimension of different of changes we might want to consider not that I would advocate yaml specifically, but blobs should be able to take anything and be content addressed. Currently, there's only one way to content address them practically and it's sha-256, but extending that to other things could be useful. I know I only have a few more minutes, but I wonder if sargon also has a stand up. Sorry.

G

My uh I would love for this to be clarified, because I asked this question a long time ago, which is if we ever want to mix hash algorithms in the uh pull process. So let's say a tag points to a um sha 256 manifest that then points to uh md5 blobs is that illegal? If some blobs are md5 and some bobs are are md4?

G

Is that legal, like none of this, as far as I know, is ratified in the spec? uh This isn't an improvement, but this is just a clarification that I think would be really wonderful to add.

A

I think it should be because each one of those requests is a separate api call for each individual.

E

Yeah yeah, I think I think practically speaking so so I agree with you that I think it's not defined and therefore is probably allowed, uh but practically speaking, what a monstrous registry to do that to you. But.

G

Like in a manifest, could you mix and match different should? Should we make that not a thing, maybe.

E

Oh, I think I I think I think it should be.

G

E

I think it should be possible right. Okay,.

G

Didn't you do it? Okay,.

B

Because, because what if you? What if you were building something on, you know, you know debian base and it was published as shout out to 56 or in the future or whatever they want to standardize on, and then you on, the top are exclusively using blade to be, and your layers would be. You have a mix, a heterogeneous set of hashes.

G

Should we make this ratified somewhere, please uh image spec.

B

Yeah yeah, it's.

B

I think this is one that would need both because, like I said, it's a different api call so having a bit of verbage around those things, the fact that you might manifest that effectively he's calling everyone to pass is fine, but then, for whatever manifests, you know that are getting pushed to the distribution and it would, I think, it'd, be in both distribution and image. Spec.

E

I think it's fine to put it in the in the specs to mention it in the specs. I think it might confuse people because practically it doesn't happen like you, you would never see this it's more of a like. uh You know, educational oddity, for instance, you could have two different types of digest, but in practicality, 100 of registries only support 256.

E

um I got to drop for another thing, but plus one to vincent's focus.

D

The the image specification does not say that you can't mix, nor does it say you can, but it does say that you there's a format I'll link it here.

B

So even just having examples in there but yeah.

E

All right see you folks have a good week.

A

Thanks everyone.

A