OCI Weekly Discussion, 15 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OCI Weekly Discussion - 2020-07-15

Description

Weekly developer discussion for the OCI from 15 July 2020; Notes/Agenda here: https://hackmd.io/El8Dd2xrTlCaCG59ns5cwg#July-15-2020---6pm-Pacific

A

B

Hey steve how's it going.

A

All right I'm going to be driving, so I'm going to be hiding because I don't think you really need to see me driving. I guess I can put out faces out the front and make it interesting for everybody.

B

That's all right.

A

You ready for round three.

B

I think so I was just trying to remember actually where I was.

A

Looking at this.

B

Dock and trying to remember if we talked about extensibility or if we just talked about lazy, fetch, basically.

A

All the way through it, but we wanted a chance to review stuff the notes or we had like one or two items left.

B

Well, so there's uh there's verifiability and uh repairability which still wanted to talk about, but I guess he's not here because he's europe um and then the last two reduced uploading and untrusted storage. uh I know the guy who suggested these and I don't think he's uh has been on any of these calls reduced up. Well, I don't know my summary of reduced uploading. Is it should in principle be free with all the other innovations.

A

Yeah, I think I commented on that one because I think there's the way. It's not it's something, I'm relatively yeah, nothing simple when you've got a community thing, but a relatively similar thing, where it's like hey this digest already exists, don't send it to me for the figured out there's some trust boundaries.

C

A

Each registry operator will decide hey, that's a layer from another registry from a crypt from a bad person. I'm not gonna. I'm gonna make you late upload that to your registry first kind of thing, but other than that it seemed pretty straightforward.

B

Yeah, I agree, or you know maybe there's some challenge off model where the registry asks for hey. What's the hash of this section of the um file or whatever you know, I'm sure, there's cryptographers, who have figured out the most secure way to do this, somebody that has.

A

History, that can say what is the problem with this, because but.

B

A

B

Yeah and then also I you know, I- I don't know that that's necessarily uh image specs thing and the content of the image is going to be the same. uh It's really a disspec thing. Is there a dispect extension for whatever this challenge off crypto protocol is or whatever.

A

Yeah, that's true. I mean it's funny because when I first looked at this I just assumed it was both um and then I realized that alex is kind of more focused on the image spec, but I kind of think of them together. So I don't not really obviously.

B

We have to figure out which.

A

One to make the change to, but I don't consider this to be just image, set, to give it as like. What's our next investment on both.

B

A

Is it just the two of us.

C

B

C

B

People 13 people on the call.

A

Then I will hand over.

D

Once I've been in.

A

My car, so I see like one icon, so I apologize. We were just kind of chatting as phil.

C

No, I was, I was bringing up the docs and then also uh so the weekly discussion, doc and then also the brainstorm use that you all kind of dove into there.

C

Assuming y'all can hear me, yes,.

B

Yeah, we can hear you vincent vincent right, yeah cool, so one that's interesting on here to me is verifiability or repairability.

B

So I you know till is the person who added this. So I I don't know that I can do it justice, uh but one thing that so you know like the blob themselves. I guess you can verify them because they're content addressed- and I assume we're mostly interested in keeping that same format.

B

I don't see any reason to deviate from it, but the repair ability is interesting. Does anybody? Is anybody familiar with the reed solomon encoding and all that.

C

B

So the the place I first ran into this is uh there's the the binary trading of binary files on using that usenet messages used to get corrupted, and so they would transmit this extra. Like 20 percent block of data, and this read solomon encoding. Basically, you know you can you can kind of tune? It there's parameters, but basically you can decide hey. I want to include this much data like 20 extra, which means, if there's a five percent corruption in the archive uh with this extra 20 data.

B

If it doesn't matter where the five percent corruption is, I can repair it um and that's kind of it's kind of cool, because um you know this could split bits or whatever, so it's stronger than a checksum, because you can actually get back the original. The thing that you wanted um in the face of you know uh you know whatever bad discs or bad networks or whatever.

D

B

And look looking at this uh little section here. He doesn't really talk at all about repairability. He just talks about.

C

B

C

B

Yeah just reading.

C

That section and didn't see what you're talking about here, but.

B

Yeah, so so this I mean literally, this just comes from like my old days of being on usenet, whatever it was, however, long ago many many years ago, 25 plus years ago,.

B

Yeah, yes, so reid solomon, they I mean it was like a pretty standard practice to to distribute repair data and you would do it kind of alongside you. So you'd have here's, here's the the data inband and then alongside here's, this extra twenty percent, that of a resolument.

B

So you could download this thing and then try and open it or render it or whatever it was, and then, if it didn't work because it was corrupted, then you could download the read solomon stuff and try and fix it, um and I wonder if there's not, because the thing is too you could. This is another thing where, depending on how interested you were in this, you could also have the repo server generate this. So you can also say, like um I have you know, 98 of the data it's corrupted in various chunks.

B

I don't know exactly where please send me a read. Solomon encoding, that's strong enough to fix my two percent corruption and you know if your repo server is willing to do that computation, then they can. uh um I think it's reid solomon. Isn't it am I thinking of this right.

C

Yeah, I'm I'm pulling down white papers right now, 1960s since and since the 1960s mit.

B

Yeah yep: this is the thing.

A

So, anyway, shopping.

B

Is missing absolutely yeah, I mean it's, it's exactly that kind of thing, um and you know when, like the last time I used it was 20 years ago. um So I don't know if that's what he means here by repairability, but that would be a cool feature to include um you know I don't I don't know where exactly because again in siri, you don't really need it. It sort of lives alongside the image spec uh or you know, I don't know if you would distribute some read solomon encoding.

B

I don't know what people think.

C

It'd be good to make that com comment that or at least um otherwise haven't annotated there.

C

D

C

I'll see if.

D

I can figure out there's a there's repairment to hampton, because I am thinking we will be paying more than we get to include this complex design.

C

I could imagine it almost being like an additional chunk or layer, that's included in there with its own media type. So it's only fetched it's it's known. Its existence is known of, and it's only fetched if needed, but the the only the only part that I think is why that would not be.

C

um I guess the reason that folks have not ventured that far is that if it's corrupted then presumably the server the server would have a backup somewhere else or would just not trust it, and if the client ever pulled a blob and not get the um same checksum than it would just throw it out and try again.

C

um The only other place that repairability to me really would apply is for the clients that are trying to push or save or export or not in the docker sense, not export, but still get the same object back off of the local graph and arrive back at the same original and we've we've we've done stuff in the past to to work around that, and that would be kind of a clever, interesting different approach.

C

But it really only applies to the client and if the client ever had issues with the image it has on disk, I would imagine it would just throw it out and refresh refresh it re-fetch it from the server. So I'm not I'm not really sure where, apart from the academic side of it, that we'd actually get into the repair ability, but it is really neat to know about.

B

Yeah, I I I hadn't, really considered it before uh jill- put this in here that maybe we should do the old school lead. Solomon thing um so anyway,.

B

Let's see what else is there on this dot.

C

All right, so what uh I have I managed to make it last week, but not for a while. So the current format here lately has been working through the the oci v2 brainstorm.

B

Yeah, basically just.

C

And then josh has a few updates on the the v1 refactor. So um we is there. Is there an amount of the oci v2 brainstorm that has been already worked through, or is there a some particular type.

B

Of yeah, I think basically, I think, last time we got all the way through lazy fetch support, um then there's so that means there's four left: extensibility verifiability reproducing repairability, reduced uploading and untrusted storage, and so I think we just sort of talked about the verifiability and repairability in my mind, um so the other. I guess we can talk through the other three and then that's sort of the end of this. um I did not.

B

These are not. These are not driving use cases for me, so I don't know that I can best argue for them.

B

I don't know somebody on the call there's this extensibility one here. If somebody on the call wants to argue.

C

Before that slightly touched on that one, the only other piece on verifiability that- and um I see that derek's on the call, but is- is figuring out how to check on a profile basis versus like with what we've done currently versus any other approaches that we might want to do or have already in place. Like.

C

Yeah, I don't know, I've got a few thoughts kicking around and no none of the none of the three thoughts that I've would think about, and verifiability of the layers on a per file basis are are listed here. So I'm I was kind of curious on the topics here so.

E

Continuity was originally developed for the uh for the verifiability case. That was, I mean that was kind of the original thing we built it for, but that's.

C

E

Verifying like the file system at any given point, reduces like the changes in the file system that are kind of irrelevant, just stores that, but it never got to a point where it's like online verifying. But it's just the way that could use to verify.

C

You know that that's what I was thinking continuity, uh the entry work, the existing and then the stuff that I was a part of and then also to some extent, this profile there's a light, a very light amount of that that's done in the tar split that is in docker and other places to check on a profile basis.

C

But it's not quite like a user facing it's meant to be just an in-between piece of it. It uses crc64, which is not cryptographic. It's just a step, a group step above what hard drives use with crc32 checksums.

C

I don't know what you're going to say for verifiability. It sounds kind of high level. The way.

D

It's written here for verifiability, I I think one one drawbank in the chronic v1 implementation is that they and it cannot be verified after the image is unpacked. So you, you feel example under the host and someone helped in and changed their the files. Then users cannot understand how cannot see it and uh it can be a security issue for some use cases.

C

Yeah and that's.

B

Right- and this is sort of why I'm interested in doing the direct bounce stuff, because then there is no unpacking to destroy that metadata, um but I I don't so I mean he mentions that he's splitting out the particular requirements separately from direct mount here. But in my mind, the way to solve this is to just have a mountable format. In the first place.

D

Yes, I agree: yeah.

C

Yeah, I agree that that you know, because if you do direct mail, then a certain formats guarantee read only b: you can do things like dm verity.

C

um If you do have to unpack, then all kinds of things go out the window and we knew that with tar archives.

C

And then you get into whether things are like active versus passively checking. So that's why several of the solutions like entry, entry and uh continuity both do kind of the view after the fact or what it should look like, um at least when the when the tar archive itself was packed up.

C

um Since some of those things go out with file systems. Tar split actually ignores some of those things to repack it. um It just uses the pure content of the file if it passes a basic checksum, I mean the other side of this is like composing stuff, like whatever ima evm stuff, um to make sure that a disk hasn't corrupted or something malicious happened.

E

I mean tar split could because if you had the actual diff id, which you could you could stream it through, basically creating the tar in check as well.

E

So it does do I mean I think it relies on more than just like the crc in the end, but.

C

It does it's: just it'll it'll fail early on the crc. That was the reason I used it.

E

Yeah, like the the point of continuity, was yeah first, it was, we knew we could do the verifiability, but the idea was that we use uh we reference everything in a strong enough hash, where you could recreate the file system. If you had a blob store referencing all of that stuff, so I think whatever format we come up with as long as it has some sort of manifest that it can keep around with the files, and you can use that to verify.

A

Did I hear manifest, keep it around with a file.

C

E

Yeah always always keep the manifest.

E

C

Some of these sections.

A

Are I do remember.

F

The verifiability.

A

Last week we did talk about you know it's great, that we have signing and all of that, but as soon as you expand it there's nothing left to verify so again.

D

A

Representative of the topic- I I just I remember that conversation and think of something we want to think about with notary too, but I might suggest just because you know josh probably has some. You know wants to talk about some stuff. We need time for that.

A

If the person that doesn't own the section is in here and somebody else isn't ready to represent, we can delay it to the time where that person is like. We can do the other time slot. If that works for the other person as well, and we can figure that out. Okay.

B

So then, let's then, I think we can drop the verifiability, repair ability and actually the last two, because I don't see um sargon on the call and then so then it's just extensibility um are either of these two folks on the call. I do not see them.

D

And yeah I am here, we uh I think.

B

D

We already covered exclusivity last week, but I can talk about it. Okay, a bit more if you, if you want.

B

If you, I think, if you think it's covered, then we're probably good and we can move on.

B

B

So I guess, I think, that's the whole oci v2, then all of the stuff.

A

Okay, I mean we can contin this isn't like a one-time or a three-time meeting. You know we can talk about many of these when they come.

D

A

Defer to josh, or what do you want to talk.

A

G

Sorry I had to unmute all the things um yeah so just wanted to provide a kind of high level overview of where the distribution spec 1.0 um stuff is I'm going to try to share my screen.

G

You see my browser, okay, anyone looks good.

G

All right so um so today, basically.

G

You know no one. No one asked to do this, but uh cloned down the uh the new open container site and added a menu item here for specs. So I don't know if this is something we actually want, but I thought that this would be an easier way to display this information besides in the pull requests.

G

So this is live here at oci blood, orange dot, io this new, this whole site and then, if you go to specs distribution, spec um and then here, uh I realized that this that hugo or or this site provides actually a table of contents itself. So I've actually removed the table of contents from the new um the in progress spectac.

G

But this is a combination of all of the prs that we have open and really just for people who have not been involved in the distribution spec stuff. What we're trying to do is simplify the spec and kind of finalize it so that it's ready to release the finally a v 1.0- and this is you know, after months and months of conversation, uh but essentially to to kind of go over what's going on here is we have an overview section with definitions and these uh these are still up for debate.

G

So I I encourage you to look at these and challenge that this is. uh These are good definitions for these things, but just defining what push and pull means. What a blob is we keep talking about artifacts, but what what does that mean? um And then it goes into uh kind of the the meat of this is the workflow categories, and this breaks down everything that was in the original dock into four distinct categories, which has also been reflected in the conformance testing. So the conformance testing is broken down like this exactly.

G

This is not showing up as a header for some reason: content discovery, but.

G

H

G

G

So um if you go to the distribution spec, we have uh all of the all of the updates to the spec, because it's such a large change. um What what? What we're doing is opening tiny, what we're calling mini updates, which are just kind of small things stacked on top of each other to be reviewed in order so number five we've already had four come in, which are, I believe, all the way up to uh this section. So definitions and up have all been merged.

G

I guess that's a lot less than the table of contents was part of that, and now I've removed it. But this is a pretty straightforward one, that's just kind of like overview of this section, but once this one is merged the we really need a lot of help, reviewing numbers, six, seven, eight and nine, and so I'm hoping five can get in within the next uh few days. I don't think it's there's a lot of questionable content in there. Six, seven or eight are really what's going to define the spec and then um so.

G

We need a lot of help from community to to review this and then the final thing. Beyond that is actually describing the the you know very low level details of the api, the actual endpoints that are being used all of the acceptable error codes and anything else, that's kind of very high detail.

G

So that's pretty much all I have um so yeah. We could really use help looking at these prs, if anybody is interested in helping with.

G

E

Get that one merged vincent and then you can rebase the other ones, don't be better easier.

C

Yeah yeah but say I think I've got.

H

Well, we have some people on that are not distribution spec, but image spec people. Do they like this idea that the specification would appear on the.

H

C

I mean it's just me.

H

And derek that are on the image spec right yeah. I I guess I assumed that people working on ocibi2 are kind of more into the details of image. So our opinions more on that right.

C

Now I think I think it's I mean having pretty ways to convey and easy ways to. Reference uh are good.

C

That was literally the reason that we had some folks in the early days ask for html, not even they didn't not even caring if it was on a website, but an html and a pdf output of it, which was a pain in all of my butt um but the having a live website, especially if it was had any notion of versioning, which might be asking a lot right now, as this is all in the works, but basically anything that gets tagged would could be its own path would be particularly great.

C

So then you could have two different tabs open comparing points in time, but.

C

Even just something rudimentary there would be nice just to have something visible, even if the website's always just the latest, and we commit some. You know you could generate it from a prior tag. If somebody cared to do so.

G

um Yeah and what like, on that point, what I just shared in the chat is the way that um we did. This was putting a make target that actually sourced it from a specific branch or tag from distribution spec. So we could lock that to 1.0, and so the website would always show the 1.0 and not the latest,.

C

uh Yeah, okay, so whatever is considered latest stable or something like that.

G

Well, or or not, I just mean.

C

I mean, I don't mean lightest is in master, I mean, or you know whatever the head of that is going to be really version of the spec right.

H

Should this be like a get sub tree or a sub module, or something like that, rather than a makefile command.

G

Sure it doesn't yeah.

H

I have a lot of people have like strong opinions on, like some modules being so I don't know, I actually don't know if this is better or worse. I guess that's not really a big deal.

G

um Yeah I mean it was just I was just playing around it. I just felt like it was really hard to show the full end results of these mini updates, so it was either.

G

Put up a website or build on top of the open containers one. So that's all that is.

A

Is your thought that this would just be on the opencontainers.org site.

G

Yeah, so the url I put in there is just my github pages build of that site. So yes, that's the idea. We have a pr open, that's in draft mode, because I don't think it should go in until it is 1.0.

G

C

Okay, no, I I think that that's fine and we chat about this recently. I just haven't worked in my backlog.

G

Well, yeah, I mean I think, a while ago we were talking about like distribution spec having its own site, and then I think it was actually jimmy that mentioned that oh, the open containers or just when open source should be built on that. um So I really don't. I really don't have strong opinions, I'm just trying to express the uh end results more clearly to people who are reviewing.

G

A

I mean just from an effort josh, I mean there's a lot of thankless jobs in the community space on these, but this one is probably the bigger one of just stocks and it's been a huge effort. I know on your part, so thank you.

G

uh Yeah I mean you're, welcome I uh it's it's. Definitely I think it's it's challenging to get people to agree on what should be in this document, because it has a lot of historical context. So yeah, it's a really good time. If people do have opinions on this to to chime.

C

C

Okay well, well, it's I'm not open tab to review that, um along with a couple other things so.

H

Yeah, I actually think this is a really good idea. The only concern is that the other specs don't also implement.

D

H

Like they don't also have pages, which then like I, it's unanimously an improvement, even if we only have one spec there. But the question right now is like somebody goes to open containers website and then goes so where the hell are. The specs right.

G

Like totally totally, I think it should have all of them and it, but what's cool about hugo, which is the um that's the static site generator for that for the for that site. um It just takes mark down, so we can actually, if the other specs are in markdown, which I think they are you just throw them.

C

How does, how does it do with just a pile of markdown like six seven different.

G

I think it does pretty well, I think it does pretty well.

C

G

All about, as far as like.

C

uh I see like the table of contents on the right side, and this is all getting into like assuming all the semantics of, what's actually in the spec carry over just fine like this was one of the things that, like links and anchors and all that stuff.

G

Well, it was it mean it blew my mind because I didn't blow my mind but, like I spent a lot of time putting together the manual table of contents and then I realized when I built it, it was like two table of contents, so.

H

Yeah, it basically generates it based off of headers, but I think what vincent's concern is more about. Is the fact that, like the image spec, for example, links between multiple documents and all those relative links are going to be broken so.

A

H

A

Their github markdown links there they won't carry over. If that's a concern.

C

G

C

Is like dot slash, config pound, whatever it's.

H

Not that it's a github markdown, I think they're, both github flavored markdown. It's just that to link between documents on github. There is kind of like a shorthand for urls that are relative to the current file and I'm not sure if that will work exactly the same hell like it's.

F

Not worth complaining about to try it yeah, it would probably yeah.

F

G

Yeah I mean I can look at this right now if we got more to talk about and come back at the end of the call, but uh otherwise vince. If you want to like sit down- and you know, try this out later work sure.

C

It would be worth a try because we did try a few different things and even the even the pdfs and things that we outputted still broke all the links. It only works on github correctly, so it's it's something to pay attention to. If this becomes like the canonical, like way that we view which you know this looks fine and great, and I would be fine with it like things, look broken on github, but they render to the final place correctly. Then that's just like a big big like touch-up job, which is fine.

C

um It's it's something we iterated through. We ended up deciding that github was the canonical like links, should work correctly on github, even if the rendered dock rendered outputs- don't have you know if they all have broken links. That was what we agreed to pretty much. So that was just the the decision point. That's all no problem.

A

F

A

Was made on hugo, so I mean, if hugo doesn't convert it properly, couldn't we just change hugo to understand how to convert the link to whatever it's output? Is this way to get up once when you maintain that github and when it's generated under you go for a website, it just generates the appropriately.

C

Sure, oh hugo is popular enough that I I don't want to be the one in the line of fire when I just broke 100 000 people's websites.

A

But breaking them on container distribution is fine, just say it.

A

I'm just I'm quite I'm teasing, I'm just. I guess I think josh is gonna experiment with this stuff.

C

We'll just we'll just coerce you until you do it steve. That's all.

G

D

G

The takeaway from this is help with uh the 1.0 prs, but also hugo is cool. So that's we can talk about the hugo later.

C

C

um I don't have any other questions right.

C

C

I think I just muted myself to say I don't have any other questions and then unmuted myself.

G

Should we drop.

A

B

A

All right thanks well so next week we're.

F

A

On normal schedule- and we will discuss if how we want to handle uh a european better time or what's the other reason of india better time india appear, which was time zones with 7 a.m, was better than the 2 p.m. Pacific. But let's go back to normal time for next week and we can from a stability plate there decide what we want to do.

C

Okay sounds good.

A

See everybody next week, thanks thanks, josh.

D

F

F