Cloud Native Buildpacks Implementation Sync, 6 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Implementations Sync: 2021-05-06

Description

Meeting notes: https://bit.ly/38pal2Z

A

Okay, so recording the meeting, let's go ahead and give um status updates. I can start. I did not write any code for the life cycle or image until this week, um but I've been keeping my eye on some stuff that came in. We participated in the bug bash for um this q. Con this week got a few prs to image, util from that um which, like in process of being reviewed and hopefully merged, um not too much else exciting. To share from my side.

B

uh Let's see, I put a pr in for some uh doing, some mocking of the player metadata restore thing, so I got a pr in there looking for reviews on that and that's basically all I've done right now.

C

I don't have any special updates to share this week. I hope that next week, I'll start working on one of the um stackpacks issues.

D

Because we were seeing some of those registry failures, I was looking at the um replacing the registry with something that wasn't docker network dependent, um as somebody saw to you over in image detail, there's a pr up for it, but uh it does seem like that. Alternative registry is behaving all differently, so still still turning away at that, I can put that on the agenda as a discussion topic. If y'all have any more uh thoughts about.

C

C

Any more updates.

A

If not, we can move on to release planning. um I don't know if anybody saw um the pr to you image util, but juan bustamante, who is a new member of the cnb contrib team at vmware, um put up a pr uh in image util to fix a bug with our retry logic. When we get an eof error, we're always retrying, even when the error is nil, so the the consequence of that is adding 300 milliseconds to every successful pull, which seems bad, um and I think we should cut a patch for that.

A

Does anybody disagree.

B

Which bug is that I didn't see it. What number is it.

C

I'll I'll post it just okay. Thank you.

B

Awesome, I didn't see it so I was curious.

A

Actually, when we can just share, we have something common to look at. um You can all see this right.

E

Probably defects pack to ramp yeah yep.

A

um I think dan correct me if I'm wrong, but I think pack is like almost ready to ship in a new release. Is that correct.

F

uh We are talking about it right now. um We might push back the freeze, etc, since we like shipped a patch directly in the middle of our last release and our scheduled current release, so.

A

I think javier said he's not opposed to cutting a patch seems like the right thing to do.

A

A

So I guess that that was the only thing that came to my mind. um That would would go in this patch unless anybody else has an idea. Look at the.

C

A

Any other things that we could sneak in, but I don't think so.

A

um All right so we're in agreement that we should fix. We should we should ship a patch release. um I guess maybe it makes sense to also talk about um the next milestone, which I guess is covered by this stackpacks phase. One issues um I think, as jesse said, we have a pr up for this one and mark t from red hat said: he'd be looking at this one.

A

A

Does anyone have interest in working on other issues? I guess.

C

I have two issues inside to me. As I said, I hope that I'll start working out next week.

A

We can maybe I mean I don't know how everyone else feels, but.

C

If anyone wants to work on one of them sure you're more than welcome, of course, I mean yeah.

B

I think the third one is blocked by the fourth one right. I think analyze needs to do the mixing information before you can start on the fail on that application.

C

B

Where is that right.

C

Can can we start working on this like at least like with some.

B

C

And without testing it like.

A

I would think so right if the format is.

C

Clearly defined yeah yeah.

F

If you sign me one of these I'll work on it.

B

Yeah, you can send me whichever one as well one of the last two that don't have a face on it. um I may not be able to get to it next week, just fyi, but um if I can I'll do it.

A

um Dan, I feel, like you, have some like good knowledge of mix-ins. Is that would you be up for this one.

F

uh I don't know about good knowledge, but I'm happy to learn. Give it to me all right I'll, be happy to.

B

Pair on that as well, if we have time.

E

Medium quality, knowledge about make sense. That's.

F

F

A

uh And jesse you said: you'd be up for a different one.

B

Yeah the validate stack ids, I think that's the last one.

A

Oh yeah, and you you also looked at this before.

B

Yeah it had my face on it once before.

A

um The only other comment that I'll make about these issues. I let me know if you disagree, but for this one validating the registry credentials, we we had reports um from from from users of tecton, and I saw in my own experience that the current way that the life cycle is checking to see if it has an off is very brittle.

A

So if you forget the trailing slash at the end of your registry url, uh it all breaks- and I wonder if it's worth like you know- maybe sneaking in a fix possible fix for that with this issue, I don't want to delay again. Stackpacks is our top priority, but while we're, while we've got that code, you know all in our context.

C

Do you want to edit this part of this issue or they want to edit, like I was.

B

Going to say add another issue.

C

B

Yeah, so someone can find it if they're experiencing problems, yeah.

C

Anyway, I I agree.

A

Okay, so and not work on it until after stack packs.

B

No, I think we could put it in like if it if it gets bumped it gets bumped, but I think it's I think it's something that we could put in the milestone. Like it's small enough to me at least it sounds. It sounds impactful enough to put in there and it's sort of related in the validation sense.

A

Anybody strongly.

C

F

Does someone want to own that, so I.

C

F

You on our notes, just like making the issue not actually doing it.

A

That could make things.

B

Like it sound like, you had experience there right now, I don't know.

A

I have and I'll talk to javier as well. I think he he also ran into this.

C

A

Might have more detailed, vlogs and stuff like that to share.

A

um All right, so this this is nice. We have faces on every issue uh which is exciting.

A

um So I guess, if we've covered anything else to say on uh release planning.

A

All right, so, if not, let's go on to needs discussion.

A

um Let's see if.

A

There are, there is one actually we should probably address this there's one spectax issue that has a needs discussion tag on it, given that rt is looking at it. Maybe we should just try to resolve whatever uncertainty might be there, and I think, if I'm not mistaken, the question was the format of analyzed pommel, but I think that's been resolved um by the spec pr. So.

B

Yeah and if it isn't resolved, we should make sure it gets resolved before he starts working on it.

A

Yeah um jesse, do you mind taking this one like commenting? Okay, awesome, thank you and um I think if we just ignore the other stackpack ones, for now, uh we were kind of leaving this one open because of the ongoing exploration around build kit.

A

I don't know if everyone saw eric's presentation last was it last thursday, thursday, before it was great? um I personally am now finding it difficult to keep in my head all of the open questions and possible enhancements to the poc that we identified.

A

So we were thinking about reaching out to eric and asking him if he'd be open to like opening a few issues on his repo to keep track of them, um so that, like discussion, more targeted discussion around specific things could be had. You know versus like this very, very long uh single issue in pack.

E

I wonder if it's worth like: do we eventually want the bill kit front end to be part of the build packs project right? Should we be creating an rfc for build kit front end.

E

I guess that's a broader group question, but I feel like that's like eventually in the end state, we want to own this room.

A

Maybe that would be a better inquiry of eric. Instead of can we open a bunch of issues like? Can we open an rsc to track and discuss all the open questions.

C

Do we want to keep this issue like labeled, as it is right now, because every week we're like clicking and then saying, okay or you're looking at this? So I'm just wondering. Maybe we can you know mark this as blocked on build kit, or I don't know something like this.

E

I can imagine exporting a tarp ball to be useful, like totally taking bill kit out of it, but it seems like this isn't just in each discussion. It's like. We need an rfc for how we would what the spec would look like if you want to export a tarball.

C

So maybe we can just change it from next discussion to natural, see and then write some a comment, but I don't know just maybe.

A

Oh, it requires our scene more formal.

E

um I could rename that if it's weird I can just pick one word.

B

Yes, I know that was the build kit example was really good. The demo of that. I really look forward to doing that. I think we've got some challenges to solve like around caching and stuff, but the uh it was super exciting to see how easy that could.

A

Be I don't mind, adding a comment uh to the last response from eric on that pac issue like you know, inviting him to write an rfc, and uh this is something that we talk about. You know maybe in like a small, I don't know, we've had conversations side conversations on our team, but.

A

Would it be appropriate to offer like to pair with him or to help him in some way with the rfc? If he's not, you know, like he's in different time zone, all that stuff.

C

E

I think there's like two ways: we could approach this in my mind, just footballing here like we either written rfc that sort of describes what we want. The whole bill k thing to look like in the end right um and then either re-implement it basically or take his implementation and change it and then contribute it.

E

um I think the other thing would be now that we've approved the rfc for component level contributions like maybe we want to talk with him, but whether he wants to contribute what he has, and you know we can throw a big disclaimer at the top. That's like this is a work in progress, and then we could write an rfc with him for how to evolve it to do all the things in the spec.

B

Third approach we could take is because this already exists. um He could we, we could create uh library methods only for this without changing the spec or the rfcs, like lifecycle doesn't have to like he's, not calling the lifecycle binaries to do the export.

B

So we could make helper methods that, like wrap up some of the functionality that he's having to duplicate um and and the only consumer would be this build kits project for now, like you know, just start supporting, you know integration through the uh through the library itself that isn't spec in rfc or anything. It might allow us to kind of experiment with with this, and he could put submit pull request against it to change behavior or whatever. If we need to be.

E

Sounds like either way. We should talk to him about, tell him how excited we are and that we would like to bring this to fruition and figure out a plan right.

E

um I know javier had been working with him, so maybe I can take action to reach out to javier and then the two of us can talk to him.

C

A

A

um I think we think we covered what we wanted from this issue, so I'm going to go back to our rfc's next standing item.

A

Does anyone have comments they'd like to.

E

Share nothing major. I was curious if any of you all had thoughts about the build right conversation we had in working group yesterday.

B

I liked the idea of uh starting to fail aggressively here so that we could come to a solution, because I also I my instinct tells me to build right is not what we would pick if we were designing it from the ground up, um and so, if we were already disallowing this, then I think we would have already come.

B

uh You know like we could add a flag in to allow that behavior, like the old legacy behavior or something like that, but start aggressively disallowing it by default. Regarding to other layers.

E

That makes sense to me. I know joe, is concerned about breaking build packs, um but I think we could aggressively disallow it uh for build packs that implement a new api like we can make some sort of cut off, so we're not breaking old, build packs, but there's no upgrade path anymore, and then you have to adapt to the new.

B

E

I feel like if we treat it like ben, wanted to treat it like a bug fix basically and just change the behavior so that we implement the spec, which says you can't write to these things and ship it, but practically.

E

I think that would cause problems that we probably don't want to do um and also, I think, the way the spec is written right now, it'd be impossible to just start enforcing it, like you'd need to do something like what's uh laid out here, where you basically tar up the layers during the build phase and then pass them to the exporter. That would require some reimagining of.

C

How this works.

B

I can't say that I've had a lot of experience, building, build packs that write to other build packs layers. It's not something that I've myself had to do, but the use case, I guess, makes sense from like a micro, build pack perspective of things that like to be co-located um or have to be co-located. I guess because yeah the invars, like solve most of the problems for anything that I've dealt with like you, could just sort of keep passing that along and append it and do what you need to do.

B

A

The problem of the build bomb seems a little scary.

A

Feels like a lot of responsibility to put on any build pack right. You know either the build pack that created the layer running at the end, like still annoying, to have to figure out what changed or you know, individual build pack contributing to that layer. It's like it's the same problem right.

E

Yeah, not every set of build bags, takes the responsibility of making a 100 accurate bomb seriously, and that's fine, because I it's an optional part of the spec right, but I do think we should set up the um api such that a bill pack that wanted to take that seriously had the hooks that it needed to do it right.

B

Yeah, I don't think any of our bill packs even produce a bomb right now, at least not a lot of them um yeah I mean that the one thing that worries me about the build right flag is that it. It basically has to be your like android sdk, build pack. That does that, so it sort of stifles integration for like a bill pack that maybe doesn't even know that that's a flag that they could set to. Let other bill packs do it.

B

It just means that in isolation, someone kind of has to let you know that they would like to write to your layer so that you update your build pack to be able to write. So I kind of don't like the direction it has to go.

E

Yeah, I agree with you.

E

The reason I sort of brought this one up as the rfc I want to talk about because well personally, I haven't run into this problem in the bill. Packs I've written, but the examples do make sense to me, um but I think the solution here, I don't don't love.

E

I feel like of the options we've come up with it's either like a bit of an ugly interface or it requires notepads to be very careful, and I don't think we've come into the best version of this yet so I'm curious to get all of you folks, who know a lot about the life cycle to brainstorm on this one like is there's a much better way of doing this, that we haven't thought of yet, because I think all of these feel like a little brittle and ugly right now,.

B

The closest one that felt right to me is slices and that's because it feels much more like docker commands doing snapshots like it's not quite the same, but that's kind of what I want is actually like. Docker snapshots like I want.

B

I want that particular you know kanako snapshot of like the layer when it's done with the uh yeah, when the first build pack's done so that you could apply it over and then you know, maybe there needs to be the ability so that that bill pack knows that someone modified something for cash busting reasons, but that sounds super complicated. I'm not really sure how you can reliably deal with cache. If someone else is right into your layer, you won't have all the information.

E

Yes, we could like cache deltas if you did some sort of snapshot like your overlaying, like you, do it as overlays and snapshots from the layer, a real like tank performance, though, but.

B

Yeah, that's the only safe way I see like actually doing the overlay stuff is like one of the only ways to like completely safely. Do it, or at least take a snapshot before still leave the code there and then like snapshot again and then like diff them like with just tarball diffs or something, but.

E

I think I also feel like they need to go in a different place like I don't like the idea of them being the normal build packs layer which really isn't supposed to be a workspace that downstream build packs, know about and like yes, environment variables can make it. So you don't need to know, but if you want downstream build packs to then like contribute metadata for their diff or something like that, they they will need to know some things about. The fact that this is a layer that they're contributing to an insured environment right.

A

I think I found steven's suggestion for uh preventing the shared layers from being exported to be compelling.

A

E

From a like, the android sdk example was a good counterpoint, though, because in that case you really do want it to be exported.

B

It seems from an audit perspective. It seems really odd to like allow someone to modify someone else's layers right because you just won't know like you know, you look at the image and it looks like it's just by android, sdk and you're like oh. They committed this really suspicious file and really it's like build pack x from like 10 bill packs.

B

E

Anyways, I don't think we're gonna get to a resolution right now, but I just wanted to get you all thinking about it.

A

I had a just one on the rfc's one more thought related to the disambiguate layer, metadata files.

A

I missed the second half of the working group where this was discussed, so I'm looking forward to going and reading the notes about what was said but emily after reading your comment, I'm scared and I don't want to do the proposed option. It seems really complicated, not impossible, but like oh.

E

The compatibility yeah comment yeah, like I think, there's no way around it unless we wanted to like. If we ever want to change our directory structures, it's gonna be a thing.

E

C

I just need to be very thoughtful about like making it now like the right way, so we won't need.

E

To do it yeah like we should not turn on this. We should make it so it's extensible yeah.

A

um You know kind of, like is sort of a like. I almost want us to align these two right. If we like end up adding shared layers directory or something like that that changes you know it, they should be compatible.

E

Yeah, I sort of started writing up that comment just to think through it myself to see if it was possible to still support all the api combinations we do now, because another option is just like we say: no, we won't run this combination vps, but that's really hard on end users, but in writing it up. It did seem like it was all possible.

A

I think, if we're confident that we've arrived at something we're going to be happy with for the long term, then whatever effort is required will be worth it. It's just like.

E

Yeah, it's annoying right.

C

Or we can just throw away compatibility and like who cares? I'm kidding, I mean no way.

A

All right, if there are no more comments on rc's, we reached the end of our standing agenda. um Like I know you mentioned the image you tilt things. Do you want to talk about that.

C

For any anybody else,.

D

Yeah, I'm happy to talk about it unless anyone wants to do something else and feel free to drop a few. If you don't want to get into the intricacies of um uh registry, deletions um drop a just. I just put together in this chat.

D

um The preface or I guess the one liner is that um oh yeah, let me get that sorry, pull it up. First, okay, here we go um the way that imageuto deletes from registries behaves differently on different registries.

D

um I'm sure this.

C

D

And I'll jump over to this, um so I put together a very uh poor approximation of what um imageytil does for image remote delete just doing ggcr libraries. This is obviously not a complete main go, but it basically takes a name. Reference result takes the digest and then makes a new reference for it and then tries to delete it based off of the repo name in the digest, ignoring the tag and the response from each registry is different.

D

So I just just doing a go around of this main: go with some more harness around it, um making a brand new image on gcr and then deleting it by digest, at least for gcr.

D

It leaves the image manifest there, um as the hypothesis, at least for me, there is that I guess that it does return a dangling, manifest tag response, um but yeah the manifest is still there, which is weird um when you try to query it by the name again, uh the ggcr, the test registry, the thing that we I'm experimenting with on that pr does the same thing as gcr.

D

Harbor does actually delete the image and returns no response, except it gives you back a not found. Docker distribution um just running a local one, not the docker hub, because it doesn't allow any deletions.

D

It doesn't return the image either, but it gives you yet another manifest unknown.

D

um So all this is a little perplexing to me um wondering if y'all spot anything strange, I might be doing uh maybe missed copying the implementation or, if, if y'all um yeah, I don't know, I'm also happy to just keep digging into it. This is just interesting stuff. I stumbled on so far may.

C

I ask a quick question: um how did you figure out that this is the I mean? How do you start working on? This did something happen, and then you saw.

D

Yeah well, um I'm! Well I'm grateful for jesse. He was doing a review of the image util pr um that I put up there and uh he spotted that it was actually meaningful, and this was the pr to change from docker distribution to the gcr registry uh for our tests. um Motivation for that has been really work around all the docker networking problems. We've had um so there's one thing that I ended up changing that test that actually was significant.

D

The response, their response, when you there's absolutely nothing in the registry, um the air response from the different registries. When you try to delete something, is different. uh It says name unknown from gcr and manifest unknown from docker distributions like.

E

That's exactly.

D

So this is just let me download that.

E

Having a spec should fix.

D

Yeah and the spec um is a little hand-wavy about it. I think I might have lost a link to it. I've.

E

Seen it inconsistencies like this across registries before so I'm not totally surprised about the uh return return codes.

D

B

Do you know if deleting from the tag makes them consistent? Is it deleting by digest that's the problem.

E

So I think, deleting by tag should be supported by most repos. um It may not lead to you deleting the underlying image or just deleting the tag that points at it and some registries don't support deleting the underlying image, but instead rely sort of on like garbage collection policies. So registry is like artifactory, like you're, never going to go, delete an image instead, you're just going to delete tags and periodically it's going to clean up dangling images.

E

um So I wonder if we should be what's interesting, is I think, the way we use this in a live cycle like just very specific to our use case, not like image util as a whole concept is that we use it for the cash image like we create a new one, and then we delete the old one, but when we create the new one, we move the tag that was on the old one.

E

So there's really nothing left to delete if you're only going to delete the tag, but on some of these registries that don't clean up dangling images, you probably do want to go delete by digest.

B

Is that is that really a life cycle problem, like I kind of feel like that's? That's your registry choice problem. If deleting a tag- and I don't eventually clean it up that seems like I mean I guess it would be a good citizen move to do it the way we're doing it, but I'm not really yeah. I don't feel too strongly that we should be doing that anyway.

E

Like we could just stop, um it gets kind of annoying in docker with pack- I guess like if you're using a local registry or something right, but I guess most people using the demon in that case.

B

You have to enable deletions anyway, if you're, using like a local like distribution, for instance, they don't even come by default with deletes enabled. So you just get errors anyway. Today, I think.

E

You can just leave it as an exercise. The user delete your image like. Maybe we just over thought it and we shouldn't do it.

D

It feels like we haven't, heard too much feedback from users. Finding this a problem right or have we.

B

I don't think so.

E

There were cases where it was broken on certain registries and we fixed those, and that was a problem, but I don't know if we stopped cleaning them up whether anyone would notice or care.

E

I guess if you end up paying huge bills for storage on whatever cloud you're using you might care eventually.

B

Yeah I'd be really curious to see if your example here is consistent with delete with like just moving.

C

B

Tag or deleting a tag, um if it's, if it's consistent, because if it.

C

B

A I'd be I'd, be motivated to just see.

C

B

Care in like a couple months, if we just stop deleting the digest directly.

E

I wonder if just thinking about image, utilize library, the way image util is architected. Now, if you call delete on an image like if it has a tag associated with it, which probably delete the time and then we can try to delete the digest but not really check the error.

D

um It's worth noting docker distribution does throw an air. If you try to delete by tag, I believe so I wonder if we should just ignore both errors.

D

C

E

I thought delete by tag was the one the only one they were all supposed to support.

D

This, at least according to the um talker's version of the bridge. I think this is oh yeah v2. um It says reference must be a digest or the little fail, and then I don't know yeah, maybe moving. Maybe there's another way to delete it like moving the tag to some sort of delete mode or something would work, but.

E

D

We could try.

E

Both and return an error, if both of them film.

D

um Do you feel like a good next step is to um start at rfc or something like that, or um certainly anything clear enough.

E

This is like a big enough. I don't know if we need an art like a rfc for this. I guess maybe.

C

Now that we have.

E

Sub sub-team specific rfcs, we could do like a smaller target drivers. Okay, I guess my only thought is. I feel like within this group. We could make this decision. um I don't know that anyone else cares about this little detail.

E

D

If your sub-team.

E

Rfc could work for them.

D

Okay and kevin, I haven't done an rfc I would like to. I can at least give it give it a shot and get used to the process um and then, but if yeah. Obviously, if the result of that is um making a pr and get more discussion on there, too, though fine with that, um does, it also feel like this is worth blocking. We should figure this part out before we change to the in memory registry or before we can answer whether we want the memory registry or.

B

Not definitely want the in the in memory registry, the speed performance should be substantial, and I think if we just make our test handle both uh like that would I would be fine with that, like personally, I think it makes it easier for users to set up and it makes it less docker dependent um for some of this stuff.

D

Awesome: okay, I'll proceed with that. I'll, um add a add some extra tests to the to the current uh pr and then open up the rfc for aerospam rfc for um changing the behavior of delete. Thank.

D

E

I can't hear you natalie. Is that just me it's.

A

Not just you all right, I have a hardware mute as well. Sometimes I need both of them and then I forget um just anything else. Anybody wants to talk about.

A

Otherwise we can joyfully, have 17 minutes back in our day.

A

A