OCI Reference Types Working Group, 17 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [OCI-WG] Reference Types - 2022-05-17

Description

https://hackmd.io/bGIxKAxPROi8KlwZMQioXQ?view#May-17-2022

A

B

Hey there josh.

C

A

um Yeah, so just to recap, I put this in the chat and um I had this hour anyway, so um I figure we can try to walk through the proposal. E forks together remember his time, um but for others on the call like, I don't think we're really gonna do any uh decision making here so feel free to drop or you can watch me type and fail and type and fail.

B

It it is a working group, it's not a productivity group, so.

A

Well, it's all about typing and failing yeah and then succeeding. Maybe we'll succeed all right. I.

A

All right, I'm just gonna, go for it.

B

So if you want to, I could also just show off what's going on in the reg client stuff, but.

A

Well, this is what I was thinking, so we can, uh I think in sanjay, correct me. If I'm wrong, the the oci playground fork should be pretty fully functional or missing.

D

I I think we just crossed a interesting conversation, so the changes we had for distribution supports the new manifest. The red client is updating the image spec manifest, and so they don't both work together. So.

A

um Well, I think, maybe in an hour we can change one of those to do that so um coming.

B

From different directions,.

A

Yeah, I I think, um let's, let's get the server up first, I think that's to.

B

Me I'll be curious to see what happens with the reference type api, but without the support for the image spec stuff, I think I'll end up breaking things.

C

All right, one second,.

A

B

A

See my screen time.

B

A

A

I'll I'll go ahead and merge this because that'll be helpful.

C

E

A

All right make.

D

You can take a look at that yaml if you want.

A

Yeah yeah I'm going to check that out.

A

D

A

Okay, you want it yeah.

D

Yeah, so what it does is it adds uh you can hit those end points. Actually, if you do underscore oci ext discover, you will see the extension api pop up. There are two more small helpers like you can see that if you go to distribution registry, manifest you'll see all the manifest that have been pushed because there's no manifest list api, so that gives you an ability to at least debug what manifests are there and what not and then we can kind of like work with any of the clients from there on.

D

If you want to.

A

So, okay, can you maybe just I mean even for my own understanding, which of these are defined by the working group stuff.

D

Only the uh um only the artifacts.

F

D

A

But this is for why just understand.

D

So today, there's no api in distribution to actually get all the manifests, and so, if you put a you, you would have no clue as to where this went until you know the digest.

D

So if you, if you call that end point uh the test, the it's there on the docks just below the second comment in your in your previous screen in the readme. So if you look at that uh to test it, you see that so it's underscore distribution registry manifest. If you hit that api you'll see all the digest of the manifest.

A

Okay, I'll have to push something first,.

D

A

Okay, all right, let's run it.

A

All right, uh okay, and then is this a good time to get reg client going or should I.

A

C

A

uh I don't have docker running.

A

B

Well, no, that just feels like a mistake right there.

A

B

Did not have docker running.

A

All right: well, let's do it, I don't care. I.

B

I'm just trolling this point because I'm you know I got to put my captain head off. Sometimes.

A

Now we'll see if zoom and docker can run and screen sure.

A

All right, okay, so I'll push like alpine in here and I'll see if this works, I don't think I have this thing. Is that on? Is that a homebrew? Can I get that on hungry.

A

What is this thing.

B

um That's, that's not gonna, be something you can google very easily.

B

Is it just already there? Is it a command that what are you saying, I.e.

D

Yeah, that's it.

C

All right, cool, stackers up.

E

C

C

You want to see what's happening here.

D

Yeah now you can just see if the manifest is there, you should be able to see the digest of the manifest. Also.

A

Oh yeah, but I'm kidding.

D

That's stalker not able to push something. Is the registry.

B

Up and running.

C

D

uh Docker running should be way.

A

It could be a dock, it could be a docker thing. I mean. Let me.

A

A

Okay, that's not that's! It's the it's.

D

Oh, can you change the data, the data location away? You want this and restart.

C

Okay, that probably that probably makes sense.

D

Right you want to put that yeah yeah I'll, just dang. It sorry about read me.

A

No, no just well, let's just like do it live.

B

I was over here laughing at that reference.

C

A

Okay, so now we're gonna do.

D

I'll just change that to alpine.

A

D

Or, oh sorry, yeah maybe I mean maybe.

A

We can add like this to the uh into the.

C

D

Yep makes sense it's good. Let's keep all the manifest show up.

A

D

Those are all the manifest that you have.

A

um And sorry, I just I just want to be clear. This is for just what why why do you have this.

D

uh So let's say you push our effort manifest without a tag you wouldn't be able to get all the manifests uh there. So you can ignore this right. Now, it's just a way to debug and test. That's what that's! Okay,.

A

Okay, okay, I just want okay, it's just for our own purposes. Okay, all right um refers api, so I can now hit this on that it should have nothing probably.

D

Correct so do you know yeah, so you can just change that to underscore oci.

D

uh Oci artifact slasher first.

D

Yeah, so you need to give it a digest, give it any digest. You want from the top.

A

uh Yeah, what is it an.

D

Argument it's an argument.

A

A

Okay, like this yep.

D

I didn't get time to make the last commit for proposal e, which has which changed the field to refers instead of reference, and so now you can copy a digest from a wow any of those and then just.

A

Do I maybe want the index of the alpine image.

D

Yeah, I don't know which one is which.

A

C

Okay, yes, sweet.

D

So now the only thing is, you should be able to put a manifest or you need to get it somehow linked. If you want to use the image spec, which I don't have the changes for yet but the other manifest it does work.

A

um Sorry say that say that again so.

D

The artifact manifest changes are there, but the way to support the image spec with the field changes are not there. That's something I didn't have uh so. Basically, maybe I'll give some background.

D

The the folks that I've uh kind of like marked as the co-authors on this they did all the changes for supporting the rs manifest. I ripped it out and modified all the manifest to support the new artifact, manifest that's oci and that's how all this thing works end to end the change to support the new manifest is not there that I mean the image. Spec changes is not there, so I think what brandon wants is to make sure that we can put an image image manifest right according to the spec and then.

A

So really quick, I have this fork and this was if we pulled this in, would that be enough? Because it's just it's just doing that.

D

This will be enough to yes to support the reference as well, but it won't show up in the reference api that change needs to still needs to happen.

A

It will uh okay, so can we do that really quick, or is that going to be pretty impossible.

D

I can um so akash is the dev on the team. Who's helped with a lot of the work. Maybe I can work with him to get that modified to support this as well, but it'll. Take me at least a week. If that's okay,.

A

um Okay brandon! Well, I just I just want to like see where we can. You know with the next 45 minutes like. Is that a good use of time or uh brandon? Should we try doing like what can we do with red client.

B

I can throw up. I can do an example over here.

B

Show you existing without the distribution stuff.

A

Yeah is it just on me.

B

Yeah I'd throw everything right on me. That's the best.

A

Are you, are you comfortable with forking this yet or.

B

It's it's already forked you're just one commit behind, so I threw one out there's a man ago, but it's already four fine. Okay,.

B

B

Yeah, so if you kick off the make, that's probably easy: what the heck are you on roach client now.

A

A

E

Like is there like a make binary.

A

It's like a make binary.

A

Is it because I'm running something on port 5000.

B

B

Very fascinating that you've got the failure over there, though um test for edge podcast.

B

Yeah, I don't think it's anything to do there. I think it's something on my side.

A

Okay, uh could I, though, just.

B

You're gonna say make binary okay.

B

Or binaries might be plural.

B

uh So skip all the test stuff.

A

I'm wondering if, um as we're going through this, I can.

A

Run through our steps, because.

A

So people can do this on them.

A

All right to make it my bindings.

A

Okay, so now what what shall I.

B

So now you've got a registry. Let me I'm just going to pull it up from I've. Got it right in the recent pr that I submitted on that one, I'm just going to drop you two commands and you can tweak those up as you need to okay.

C

B

Because the formatting okay.

B

Okay center video, on slack.

C

C

Change these two yep.

B

For the repo, after that, as well, you've got like a alpine reaper name that you were using.

A

All right, maybe I can like annotate what what these are doing. So this is creating a using the new manifest.

B

It's creating it using the um the refers api if possible. Otherwise it falls back to pushing it via digest tag.

B

And that's going to give you a listing.

B

It is going to look for that repo column b1, so you need to make sure that image exists.

A

C

Go ahead and change this.

A

That is that gonna be okay.

B

B

Now, the one thing I don't know what's going to happen is, since we got an api on your registry, if that's going to cause some chaos, so you might just want to spin up a regular registry instead of the the fourth distribution district.

A

uh Well, do we do we want to see if there's any chaos.

B

We can see if there's chaos.

B

Oh, I need to give you another command there. um You can control see that it'll there it goes. uh I need to give you a command for.

A

B

Yep, it's fred, ctl ctl registry set dash, tls disabled. I believe, and then you're going to have your um local five thousand.

B

I think that they got a century was gonna. Do it that's all from memory, so I've done work just playing my memory.

A

There should be like an equal here or.

B

B

um Okay, so go back.

A

There's like a global.

B

No there's there is a config file that we're setting. So it's not part of the artifact put command. It's just a reg ctl registry set command.

A

B

Let's change that to like your ben slash, cto and then get rid of the tls on the other command.

A

Okay- and this.

C

Was like something in my.

A

Home directory yeah.

B

It'll drop in your arm directly.

B

E

B

Got some failures.

A

B

We got something: oh um I'm looking at too much, you know. Aren't I well what was all that.

A

Well, this is the other end of it.

A

Okay, so we got.

B

A size invalid, which is curious.

D

He's trying to retrieve gcr's access token on the client yeah.

B

It does that, because.

D

It's parsing your docker.

B

Config, oh, that's just a warning: okay, um the next one. After that, though, I'm worried about when it's going to the trying to send the trunk there, you can throw it in verbose and see more details. You want to dash v debug.

B

And so we try to send that upload location, it didn't like us, and then we tried to a patch request and it just said bad request, which is even worse.

A

Is it possible that the fork is throwing them here.

B

Provide link to not match content makes me wonder if I'm missing something. My code would be the first time.

A

All right, let's have some fun.

B

Here it worked on my machine.

A

uh All right well here, let's look at the diff of distribution.

D

Do you do you just want a test client to play with I mean it's, it's a total hack that one that I used to test. It would.

C

A

um I I I mean sure I think it'd be cool just because efforts like are going on this client from brandon and then you and your team is doing this one, but yeah I mean you can we can try that as well.

D

No, that's why I didn't want to kind of like bring it into the picture if we can get rich client to work and that'll probably be the best thing.

A

uh Okay, but let's.

A

I just want to see.

A

So brandon you're testing against a distribution like release.

B

Testing is it yeah the registry calling too.

A

Like if I were to do like yep, something like this.

B

As long as the port is open,.

A

Okay, so I gotta push again.

A

C

Okay, so that went.

A

Yeah we need to see, but this this did what the fallback, because look, this is doing a 404.

B

Yep, so we get the that didn't, go and so pushed by the digest there, and so, if you now go back and run the other command for the listing.

C

B

All right back for the reg client, so we had two commands in reg client.

C

Yes, sorry too many windows.

B

And so you list three first now it'll tell you hey, there is a referrer there, which is the one that you just pushed right now.

A

Can I get like a jq type thing.

B

um You can say dash format and then um what are you looking for some json? Pretty you can say the format is going to be a go line template. So it's a single quote: double bracket.

A

B

B

Yeah and I think I've got jason, pretty dot.

C

That's okay: let's see.

B

Either way it does exist, you can also say a tag ls on that repo and see the tag that shows up.

A

Okay, this is nice.

A

So is this your own.

B

I went ahead and made my own. I was trying to figure out how to pass this data structure around things.

A

And then, but is it? Is it gonna accept the response in proposal e yeah.

B

Everything under the covers is image is going to be a is an index under the covers, so that it's perfectly fine, it's just when I start passing things around from piece to piece within my own code. I just made my own thing. If you.

E

B

Said uh dot manifest with a capital m you'll recognize what you got.

B

So do that same format, real quick, um where they're pretty just after the dot say manifest with a capital m.

B

That's what you're, looking for, which is just an index.

A

So this is the pure response, as as.

B

I regenerate it to look as if it was.

A

Okay, I don't I'm not sure. I fully understand what this format thing's doing, but it does. It does seem to do exactly what we want.

A

All right so now, basically, what we're looking at is the the no updates registry right and it was pushing to.

B

A

B

Directory in that repo you'll see, the tag shows up.

C

A

C

A

Latest in your digest, so let's figure out why it's not work so so I think it might be an issue with the client then, because this.

A

Well, I've got to copy it in.

A

And that was fast because I have the temp directory. I think all right what's going on here so.

A

You have, it should hit that end. Point first,.

B

You know when I push it, I am going to try to hit the repair endpoint.

A

So it goes head on latest.

A

A

Is it because there.

B

Go over to the other one. We got debug.

A

Does this need to have a different tag? Probably because it's it looks like it's like doing a 200 uh on the head.

B

I'm getting the digest for alpine latest because I need to push my digest. I need to push the reference as a digest. I can't push a reference to that: hey, okay, so.

A

But where is the shouldn't here? You should be checking for that underscore oci.

B

A

For that comes in I'm not.

B

Seeing it so that's where I'm thinking, let's get some debug log on the reg cuddle and see where he's talking the registry on its side.

B

So, let's see we do ahead, then we do a post on the blob upload, so it just went straight to upload there.

B

um I guess because I had to push the blob first and then the blob failed.

C

B

I haven't even gotten to pushing the manifest because I need to push the different parts of the article. Oh.

A

Okay, um I see so maybe there really just is a size.

B

Issue, it's you know it's going to be a small, it's quite possibly because my config is empty.

B

When you see which blob it is, it doesn't like, but I've got an empty config that I'm pushing up that doesn't have it's just got a couple brackets in it.

A

It is this: is this like a header, we need.

B

Which are you in right now, you're, looking at the artifact puts in reg client bridge cut off.

A

Okay, so is this what you're doing.

B

That that's the config right now yeah! Okay, unless you.

A

B

One, unless you give me one.

A

So, okay in in size in 64 config, is that.

B

Should be a valid, but I'm just wondering if the registry doesn't like it for some reason.

B

No don't do that. Okay,.

A

All right that looks correct.

C

B

Was cracked enough for my fork, a branch of distribution.

B

But I wonder what I broke for the fork version.

B

What do you think sachi anything that would have changed validation of blobs?

B

You did push the config blob right before you put them out, I'm trying to push the config blob and I suspect that might be the one that's failing, but I need to look at the debug log a little closer to know if it's the debug log or one of the other blobs.

A

uh I don't know all right: let's.

C

Wait a second, the error code, size.

D

D

A

D

This is all about a mean from distribution. None of this should have been changed, because these are standard blob, upload parts.

D

What could have changed here? I.

A

Think there could be something that brandon is doing, that the spec uh talks about that distribution doesn't do. I.

D

Do the other? Oh this, because we're using emit spec, we need the config blob right.

B

I still send the config blob.

A

I'm gonna just do something bad here: real quick.

C

Oh all right so that that's.

B

So, what's this.

A

A

So now let's try the list.

B

C

What I was kind of expecting.

B

That that actually makes perfect sense to me because we'll push it um as a reference and so we're assuming that api is there we're pushing up there without any tag. So there is no new tag created for it. Yeah.

D

B

D

B

But the registry isn't going to parse the refer out of the manifest, and so it's not going to list this.

D

um So quick question: can we just check if the manifest did go through? We can actually see the manifest right. What got pushed you have the.

B

Http command. It's it's up a couple lines here. You see the push by digest.

B

You can also query that manifest api yeah. We can just directly view the.

B

Manifesto course, just without r.

C

B

um uh Just throw a header on there.

B

Or honestly, if you know that much of the url there you can just do a manifest get within um regicotto has a manifest kit command.

A

But this this is what index I should be putting or I'm sorry. This should be. The new artifact kit.

B

um No, this is going to be a standard image spec manifest, so you can take that digest and just do a reg cuddle manifest get localhost 5000, slash, oci, playground alpine at shop, yada yada.

A

E

B

And I'll have to strip out the b2 in there.

B

And then, at the end of that at shaw on your digest.

B

There you go well again with my pretty formatting sorry.

A

B

You can uh just say um format here, just say body: you don't have to putting anything else in there, just all lowercase body, nothing else. No other.

B

Take out all those quotes brackets everything else, capitals, dots.

E

B

A

So wow, so we did it done. That's amazing! Okay,.

D

The last part that we can do is kind of stitch. The uh the refers part to kind of like make sure that this manifest also returns from the in the list of refers.

A

B

Which means image spec needs to have the field in there within the registry, and then the registry knows to need to know to parse. It.

D

Yeah, because that changes the image spec parsing, that's why we didn't get to do that.

A

uh Hold on sorry, so we need to pull in the new the forked image spec. Yes,.

C

A

All right, let's, let's just let's start there,.

A

uh I'm gonna leave is that, okay, if we leave this.

B

I'd like to know what I'm doing funny here, that's throwing this off.

A

A

C

C

B

That's what happens when we do it live yeah.

A

We really we're really doing it here. Okay,.

C

I know I don't like oh okay, I guess.

C

C

Go go, go, go all right.

A

So so what now so.

D

So uh the next thing that we probably want to do is this is a so how we wire this up is, if you look at uh you have source, let's jump to source, I can maybe kind of walk you through uh if our caches that he can walk you through as well.

D

So if you jump to the extensions folder inside distribution.

D

There's a registry under registry extensions oci, and then you see artifact manifest handler yep. So this is how we uh kind of like stitch the um like new, manifest type, and it puts the manifest into the storage blob and saves it on the file and things like that right now, this man, this handler, is called only when the media type. So this is how it works. You see the media type of the manifest distribution, then kind of like stitches this handler to that specific media type.

D

That's why it's easy emit spec handling happens in a different place in this code base. I believe it is manifest handler.

D

You can use under the handlers itself.

D

Yeah, so here is where you can actually check. um You can scroll down even more so you see the media type, uh it checks where schema is and then so forth and at the end of it you can.

A

Might be misunderstanding.

D

Right, no, that's yeah! That's where the forking happens. Basically, it checks. The extra headers then looks at the media type and things like that and it handles the image manifest here. Basically,.

A

uh Okay, so it it gets the media type and then we're down.

D

So this is the get. Can you look at the put path? That's where this thing needs to happen right, there's a foot manifest as well. I think.

A

D

D

So we will have to modify this to understand that it has a reference field and then call the storage handler to put that in somewhere. um That's the change that needs to happen. Oh.

A

Okay, so that's going to take a bit to like.

B

A

Actually put it in and out of, storage, okay,.

D

So uh akash, I don't know if you're on the call um yep I'm here, do you think you can wire this up quickly or is it really um going to be hard.

F

uh So just wrapping up on this, I'm still a little bit confused on exactly what's.

D

Yeah, so so the the current approach- maybe we should talk about, is that in the image spec there is a change to support the referrer property, uh not using the new media type. So what we want to do is let's check your referral comes in. We want to call the storage handler to the reference call the reference list. Also, that's the change. We want.

F

Okay, so to put if the field exists, then we want to put it in the reference. Okay, um yeah. I think we could. That's probably the only place the change needs to happen, but uh I'll take a look at it. Further um yeah.

A

Awesome, uh are you a cash? Are you on github? Can I put you on? Yes.

F

Yeah um so yeah akash akash s- I oh, I guess I haven't contributed here. So it's not popping out.

A

You can throw it in the zoom chat if that's.

F

Yeah, let me type it in okay.

A

C

Maybe grab the path to this really quick to help out.

A

And then maybe for the last five minutes, you can summarize what just happened here.

E

A

All right, amazing uh and then here I'm just going to pull up the hacking d and, let's maybe just like summarize what we did here, so we can share with uh alicia, jason, locky, etc.

C

A

uh I I, what did we just do here.

B

So, on the reg client side, we push to manifest using the refer api, but that is just pushing a manifest. There's really nothing on there for api there. So.

A

Better said, we pushed it using using the new um artifact type.

B

We pushed it using the image spec artifact type, with a new refer content in the manifest.

B

So image, spec manager and.

A

We also, we also had it working on.

A

A

What was the thing where we were running docker off the shelf.

D

Registry distribution right.

A

Right we had like the fallback working I'm trying to understand. That's so we were able to push it.

B

Are you on the right, hack, empty.

A

B

I was like I'm missing.

C

B

C

He's typing things.

B

Thank you very much.

C

And modifying notes from last week's.

A

All right so really quick, so maybe maybe we'll start with um red client on a fresh run registry 2.

A

We successfully push the manifest with the fallback.

A

A

uh Pushed manifest with.

A

Oci playground, distribution.

A

B

I'm going to do some looking on that error, I think I saw a commit to change those lines and I'm guessing a commit has not been pulled in with a release.

C

A

B

A

B

A

A

um And then the missing piece is: oh, I'm sorry.

C

D

Yeah go ahead. Sorry.

A

I'm just I'm just trying to remember everything we did with um with the distribution piece before we started playing with red clock, because I remember we saw the refers right.

D

A

So what did so? How did how and what did we do, maybe just say words and I'll put them.

D

um So uh we can start with we gotta implementation of distribution that supports the referrals api using the extension api.

D

uh Does not support the current implementation does not support the image spec changes. So that's the other thing that you need to see.

C

All right, let me just put a link to at least two.

D

The other thing, maybe we didn't discuss uh brandon there is a repository inside oca playground with uh with the artifact spec. If you want, you can just hack on top of it or just upload the specification into that, whatever you want to do.

B

Yeah I'll get something to add. My repo is just not going to happen today. Okay,.

D

C

This is all I think.

A

Yeah, you can delete that. That's true, okay, so.

A

So reg is red, client missing. Then the new artifact type.

B

It is that's massage, it's like you get me to add. So that's a tbd.

C

B

B

Oh well yeah, but you're on the floor. You can throw that right in the main, like I said, throw it all right in there wait what I don't mind, throwing issues right over in the main.

A

Oh well, sorry, like my timing, is I'm running out of time here.

A

Okay, so on the client side, just to summarize in the client side, uh I'm going to stop sharing so on the client side, buy a red client, we're missing new artifact type, otherwise 100 percent there on the 95.

A

And on the distribution server side, we are missing um the uh support for the middle tier um artifact push, which is with the.

B

A

There's some storage work that akash may be helping with to get it in and out in that format: yep. Okay, amazing! uh Well, it's exactly uh 2 or 12 or whatever time it is for you. So um I will I'll update the slack with what we did and then maybe you all can comment with corrections.

B

Sounds good thanks for driving.

A

Amazing. Thank you. Thank you. Everyone bye.

B

Stuff quasi works.

A

It does we can tell people now.

C

All right, bye.

B

See you next time thanks. Everyone thanks. Everyone.

B