Cloud Native Buildpacks Platform Sync, 3 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Platform Sync: 2021-02-03

Description

Meeting notes: https://bit.ly/38pal2Z

A

Perfect sorry about that, I totally forgot. This meeting was happening.

B

Too busy releasing pack you're right outside.

A

Wasn't me who's dad, nice? Okay, you guys just start off anybody. We people can remember to sign into the doc myself included. That would be great um chord, meaning anybody have any status updates relevant.

C

uh Yeah, I guess a new version of pack- oh 17 0, is just cut, um it does look like some of the homebrew stuff might be a little wonky, but so I'll be fixing that today.

A

Sorry, that's probably because of m1 stuff right.

C

Who knows: okay,.

A

But otherwise, okay, great, I guess we'll get into the release planning stuff um and you also did work. You have a uh draft pr for platform, api, six right.

C

A

Okay, so that helps for the life cycle, folk people- um I did a whole bunch of small pr's for acceptance, tests and a bit of dachshund within that, and we adding um m1 to the trying to add m1 to the release. At least the binary should be on the the pack release and yeah.

D

I have a quick one in regards to techton, so I just finished: creating a test: workflow github action for the techdown integration, repo uh via kind right. um The only thing it looks like uh github actions are having some issues right now. Their performance is degraded so waiting for that to complete and then merge that in and go on to the next item.

D

I think one of the things that I want to maybe discuss among this group. I'll add it to the agenda is the platform api versions for techton what the strategy there should be. I have something in mind I'd like to propose, but we could have a larger discussion.

A

Okay, fantastic um next, I guess, is release planning or release discussion, um so you said dan went over well, any of that was the homebrew pipeline, the only one that uh that didn't work for the release.

C

Sorry uh tbd some of the linux ones just finished their like own binary building step, so I'm gonna go and verify all the ubuntu ones and then I'll do the arch linux one. But I've just been poking around at home right now.

C

A

Okay, um so, based on our needs, discussion um our comments last week, so um I you know pre, I prepped a few things that I thought. Maybe we should have been need discussion for a while, um but how about? Let's skip that? Let's first do other topics in the working group because those have been around for a bit and then we will go to the news discussion as we have time. I guess if that works, for everybody.

A

Great javier, you want to talk about the tech time platform, api version.

D

Yeah, so I guess, as we're talking about um you know, adding support to something like pac. I think it makes a lot of sense and in kind of keeping as close as possible to the latest and greatest version of the platform api.

D

But I guess I have some concerns about how we would be able to achieve that for something that's a little bit more complex, like tekton, because, obviously like it's a essentially a whole different uh task, right per uh platform, api version. So if you think about as we're iterating through all of these, um it seems very painful to have like let's say right now. I guess probably want to have three different versions: platform, api 03 to 6, and so that's three versions and we have two tasks.

D

So that's like six versions right and then we're gonna have platform api. uh The next iteration of platform, api! Sorry coming up here, it seems like we release, maybe every three months and uh so yeah like I feel like it's gonna snowball into this, like maintenance, hell uh that um and then the other aspect.

D

I guess that, like I, maybe want to throw into this formula trying to figure out exactly what to what to use is how fast do the actual builders update the lifecycle that they're, using um with that platform api right and whether or not uh we we should be taking that that into consideration to determine exactly what we should support from a tecton perspective, any initial thoughts, you know, I guess dislikes there.

B

Also, I think the heroku builder is it's not always up to date on the latest lifecycle. um So I think you know there.

C

Would be some concern.

B

About lagging, but I don't know if the tecton's unusable, but I mean we could also get better at that.

D

Yeah is there a reason why um you know other than like don't change stuff when it works kind of thing? Is there a reason why it doesn't get updated as soon as life cycle releases? We.

B

Just don't have the automation for that built. In I mean it's just sort of I just haven't, haven't really needed it much. You know, but we should probably work towards doing some of that.

C

Yeah I mean it also, I guess, when we were doing a bunch of that work at the time, it also would break our stuff back then, um which I think is part of why we didn't invest in automation,.

B

And so is the proposal here that techton would like interrogate the life cycle and the builder to figure out what the latest version is to run like. How would that work? Because that's what pac does right.

D

Yeah see that's the thing like I don't know like everything like the tecton tasks are pretty dumb right. We've often called it like a simple platform or a dumb platform, and I can't think of a proper strategy for the execution of the life cycle to essentially change right from one task to the other. Yeah.

B

C

B

Questions here like right now, I think we're still defaulted to whatever the default of life. Lifecycle is when it's not specified, which is you know, 0.3, which is quite old in platform um on one of the platforms, we're working on.

C

Because we just don't set it.

B

Because we don't know what the builder is going to be, we haven't done the work to like interrogate the builder to figure out what version like pack does.

D

Yeah like if I'm looking at the builders right now heroku is supports platform. Api 05, as the latest google's builder, supports o4 right, like I think, like they would probably even stay further behind um as well.

D

And it seems like we got o6 is that the latest that's coming up so right.

B

Yeah, it's being worked on.

D

So, like by initial, like my instinct right, is to support, like the the majority supported uh platform, api and just kind of stick to that until we find a better solution long term so right now it would be o4 and as far as I'm aware, like the lifecycle, will never break that compatibility.

D

So it would just be newer features that might not be.

D

Available, it's kind of it's not ideal,.

A

But would it be like an explicit n minus one, or is that.

C

D

I guess the problem: if, if we become an n minus one, it goes back to the argument that we move fast enough or we iterate fast enough in the life cycle that essentially it'll grow to a pretty big extent. Like I'd rather focus. My efforts on supporting the two tests that we have and adding pipelines so like pipelines, are something that you know would be nice to have, or you could just pluck it from the catalog, and it would have like a fully working workflow for individuals.

D

So again, that kind of adds to the number of things that we'd support so I'd for the time being, I'd kind of be against an n minus one situation and just be a majority supported uh platform, api version, and then I could do some research to figure out whether or not there's some sort of uh dynamic solution within tecton. It's going to.

B

Say it's going to get kind of wild like because I think in either 0.7 or 0.6, like potentially switching the order of the you know of a detector and analyzer like that, would be a big one that you couldn't just suddenly support and not support at the same time, without two different templates and something like tekton right, yeah, I'm not sure you're. Just using creator.

D

Yeah and that's the thing like we have a creator workflow, but then we have the individual phases for the security aspect.

D

So anyways, you know just throwing the initial idea out there uh with the current strategy, but I do think we probably want to solve this. uh You know in the near future, because I'm sure this is just one pain point and other platforms are going to fall into the same pain point.

A

Interesting makes a lot of sense um that could tie into our tekton version. Also, maybe we can version the task based on platform api, I'm, although that you may have to I guess, for uh for with platform or six with the switch from switch for binaries, but.

D

So the way that the tecton catalog works um is like they're they're treated as immutable versions right. So if you wanted to apply a fix or additional features to, let's say platform, api 04 you'd have to actually bump to o5, which you know which collide with that. So I don't think that would work. Okay,.

A

I wasn't sure at this point whether they were all in unimutability but looks like they are: okay, fine,.

D

B

D

But I'll see what I could find this.

B

Does seem like it's going to be a problem like just the reorganizing of like the execution steps like like you could never really write something that would support both like easily like. If you just support every builder out there, like you, would at some point have to choose to drop like platform before that right.

A

B

Yeah, that's only for trusted scenarios right like if you don't trust the builder, then you definitely don't trust them with your registry credentials.

A

It's definitely fair.

A

um I think next thing on the agenda. If this is done happy to stick on that for longer um was the windows sid rc, I said micah you wanted to talk about that.

E

uh Yeah and I get a pretty good um sort of detailed internal implementation explanation in the other sub team, so I figured I could talk to this from sort of a platform perspective um and not quite go quite so deep into that.

E

If that's useful, but I did have some particular questions around how to treat platform api uh versioning with this potential change, um the just is feel free to click in, but the verbal gist is that in stack images stack, build images will stop using cnb user id for windows instead use cnb user sid, and then their group equivalents as well. Sids are like strings, but they're meant to express the same thing as uid, just a unique identifier for a particular user on an image from the platform perspective.

E

All that really changes that you have to validate that like when you're creating a builder from that build image. You'd want to make sure that that environment variable is the windows version on uh on a windows image, and then, when you call the lifecycle, you'd call the lifecycle with a different argument rather than dash uid you'd call it with dash. U s id still like the same analogs all the way throughout.

E

You just use this different identifier for each windows, user, as opposed to the linux user and also from a platform perspective like platforms like pac that make build packages.

E

Builders, those types of things where you're, applying specific user ownership on files that end up in those resulted images um you would need to. Rather than setting like the tar, header.uid you'd set, the tarheader.pax header, dot, sid or raw sid, there's more more grey details in there. But basically you just stop setting the tar uid and start setting this window specific equivalent, um so more and more details in the rfc itself.

E

um But I'm kind of wondering, since this changes both to incoming build images uh incoming builders and how we call the lifecycle uh binaries that are inside of there seems like it kind of maybe creates a unique kind of api bumping uh situation.

E

um I'm wondering first of all how much you all care about windows, image changes, since it's still experimental, if it warrants both a life cycle version pump and a platform api version bump.

E

If, yes, then, how much backward compatibility we might want?

E

um How much we want to support sort of old formatted builders and new platform, api platforms or all those kinds of things or if we just want to say windows images are experimental.

E

Let's just say old images, don't work anymore, make them all the new way and any uh yeah. I don't know if y'all had and there's a lot to kind of dump on there, but I don't feel any thoughts on that.

D

I think, from my perspective, um anything that changes the spec is thereby a change to the api right and and because the apis are pretty. I guess you could say like isolated in a sense, um I don't think that we talk too much about like compatibility from that aspect right and so, when we talk about a specific builder image, I think we could talk about what that image is is really supporting.

D

But I guess maybe what you're discussing is like the fact that the life cycle itself supports multiple platform, apis right and and how that really affects this particular or how this issue really affects. That particular.

E

Attempt to support that, I think so, and and mainly just so, I can codify it in the rfc to talk about it, but if it would actually be more useful for me to not answer that question in the rfc text and leave it open for discussion in the rfc comments, I'm I'm also happy to do that. If we want to have more context when we talk about it but yeah, it is around that of sort of the intercompatibility and intercompatibility of the life cycle and supported platforms, uh intersected with windows, images being experimental.

E

um Yeah, so would you say, is that worth uh discussing now or should I try and make it uh discussable in the rfc.

D

um So I I think from like a platform perspective, we're kind of at the mercy of whatever the implementation team decides on here.

D

I feel like it's really more in their arena of what they're willing to compromise right if they're saying that a version of the life cycle can live inside of a builder that can, you know, be essentially cross compatible between different versions of platform, apis, it's really up to them to say like oh, you know what, in this particular case, we're going to have to break that contract, and I think that discussion is is better suited there uh because from you know again the platform side of things we could probably like work with both of them right like at least pack right.

D

It could do something very dynamic where it could essentially be passing both the tar, headers and also the other thing right, depending on which environment variables it has present and stuff like that. But it's really the implementation team that decides whether or not they want to provide that level of backwards. Compatibility.

E

Yeah that makes sense and I'll try and frame it around. That too, um my suspicion is that pac itself, uh thankfully I think as far as headers go can just completely switch over to doing it um doing it. The new way for.

D

Building packages.

E

And that yeah just right.

D

But they have that information, though right, like that's the thing. So if yeah, we don't have that information, and that's where I was gonna, ask whether you know it. It warrants new environment variables or if the current environment variables could be overloaded to represent sids when you're talking about uids right. But I think that's a rfc conversation.

E

Perfect yeah that makes sense cool awesome. Thanks for the feedback and I'll um update the rfc. With the context there.

A

uh It's funny I mean I guess we always see this, but it's always funny seeing how we kind of assume everything's ultimately compatible, and it's not always so simple. So thank you for putting in the work for that um a bunch of issues for needs, discussion, um one I thought dan may have some opinion on, and that is that we recently in this release did some work on caching, specifically allowing people to have or to publish their cache images um and related that we also talked about uh that was kind of related to volumes.

A

So there was a an ask from a user um uh to define cache value names. I think that would be based on a based on a flag and passing that passing those names to the life cycle. I guess no nevermind.

A

um It had a uh a discussion needed tag for a while. I was wondering whether anybody had first of all remember seeing this issue and whether anybody had any opinions on it.

D

So I think emily was initially kind of against the idea, um but I think the the need for it has come up more and more, um but the alternative. So I guess some context. The alternative for this was that uh the users are able to do uh like uh volume melts right to wherever they want uh somewhere like m2 cache, you know and and just kind of know what is behind the scenes of the buildpack implementation of where that's going to be.

D

um But in this particular case, because we've already provided cache image, where they're able to kind of plop in an image for caching certain things and use that cache image across different quote-unquote applications right. It seems reasonable to allow for the exact same support with volumes, and so I kind of I'm in the stance where we should be able- or we should allow the users to be able to kind of enable that functionality.

D

But it not be the default right like it's more or less like you know what you're doing and you understand the risk of essentially which is cash poisoning, but they again can do that right now, with cash image.

A

D

So does that I'll somewhat answer or give you some context.

A

So you think that to go pretty much and it's definitely do we need to do anything um in terms of we pass the volume caches and are they they are past the life cycle right so do we need to? Is there any um change in how it's done that we would just pass in the new names.

D

Yeah I mean, I think it should be very simple. I think it was controversial at first, but I think once we remove the controversial aspect, it's a simple ask. um I the only thing I'm thinking about it's, it's the whole directory right so like I know we have different um different volumes that we use, there's like a launch cache and then there's the other uh volume.

C

I don't know if.

A

C

A

That built in that launch.

C

D

So if anything, it would be a prefix, but I don't know that we actually need to provide both of them to be configurable, because if I remember correctly, there's only one that the life cycle accepts the cash dair.

A

Okay, all right that makes sense um the second so less discussion there now. um The second was an ask from natalie about pac, preferring lifecycle, api versions over released versions.

A

Should prefer lifecycle api version over release version.

D

I completely agree with that. Okay, um the only thing.

A

You'd be assigning a default lifecycle version at all. Is that something people think makes sense, or we would there would be some integration with? uh Would that would that remove um the pack packs knowledge of a default lifecycle version, or we would just assume that it's more, that um they would pack would still know a default lifecycle version. It would just prefer talking about lifecycle versions, uh lifecycle, api versions.

D

It's really platform api versions right, okay, so it the idea here is that somehow it should communicate with, like the repository, the lifecycle repository and the releases, and the releases should have enough information to tell us what platform api version it is and then uh pac should know what the I guess latest supported platform. Api version is and then essentially select the latest there as well.

C

E

D

More or less a strategy.

C

Yeah, just just adding- because I I think emily had said um now- that the life cycle is making this sort of backwards compatibility guarantee. You can always just take the latest life cycle because for any given platform api. Yes, it will be supported right. So.

D

um That is interesting. It does definitely simplify it, uh but.

C

No, I guess that should work.

D

I was, I was thinking about jesse's uh comment earlier about there, potentially being this upcoming case where the live cyc, the orders are going to change, so the should be a harder break, but I think this still would work.

B

It work if the builder sort of intends to ship their own life cycle. It only works if we're expecting them to use the exact lifecycle we distributed right.

C

B

C

B

It because it's not really sp yeah, if they're expecting it to be 0.3 and they ship their own life cycle or something. Then you try to execute something past that it should.

C

Work according to.

B

Spec but it may not work the way the builder intended, but maybe that's.

B

D

Cool, so I think for the purpose of this particular issue, though I think what natalie mentioned this makes sense is there's no reason to inspect the lifecycle repo and by default, just pull the latest version when creating a new builder.

D

Obviously, the user still can configure the builder tomml to specify specific versions or specific uris to pull life cycle. Binaries.

A

For that fantastic um then the last one I actually know nevermind we are at time. So I guess with that everybody thank you for coming and I will see you next week or later today. It's the case maybe see ya.