Cloud Native Buildpacks Working Group, 20 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Working Group: 2020-08-20

Description

* Stackpacks

A

A

A

B

Both emily and steven said they were going to be late, so we're going to give them time.

B

A

Should we get started sure it's ten past.

B

uh Yeah, should we do all the please sign into the doc new faces easy today.

A

uh Yeah signing the doc, um I recognize all these faces, so I don't think there are any new.

B

All right, uh you ready to talk about stack packs. Let's make all the decisions before stephen gets here.

A

I thought that was the plan. I.

C

A

That's why we were waiting.

C

Because I feel like we keep making decisions without everyone there and then they just go around in circles. Do we.

A

Just need a scheduled meeting where stephen can show up on time.

C

This is that one supposedly.

B

It is a full-time job, keeping emily and steven happy. At the same time, it's really difficult I'll, like I'll, get advice from emily and then do that and then I asked steven to look at it and he's like oh. Why would you do that.

B

And vice versa, by the way.

C

I feel like we just need to argue with each other up front and then come talk to you. It's probably the better way to do it.

A

I feel like that's. What we used to do right is that you would go off and somewhere and then you'd come back with a unified response. Yeah.

C

Because, before stephen got too.

A

C

Now and steven be like whatever you say I'll agree with, but then it's doesn't drain.

A

C

Or like right away, it doesn't turn out to be true, I feel like he could be talked into a lot of it, but it's still a process.

B

I think uh some of the things we'll talk about today, um the the challenges that this for me just I don't want to copy-paste a lot of code. um You know I just want to structure it so that it's you know like if we're reading the platform directory or whatever we're doing at once, the same way, but then the other part of, and that part's easy, because whatever we'll figure that out, um but the other part that might be challenging is, I think, like certainly for me as I'm working on the implementation for the stackpacks.

B

I want to err on the side of just. Do it like a normal build pack for as much as I can basically every decision, that's where I start, but I think that's not my impression. That's not where steven's starting and he's thinking much more of it as like.

B

You know like from first principles like is an independent concept, so when it comes to like launch toml we're not quite seeing eye to eye. For that reason, that's is there.

A

Just for context, I guess um what are the things I guess backing up in case people actually watch this recording. uh What are the?

A

What is the topic, and what are we hoping to kind of get done in this? I guess yeah, it's a good question.

B

Is it are we trying to get the rfc to the point where it's everybody agrees on it, or are we going to talk through some some of the code in the poc, because I think that's been the point of contention more than anything.

B

um I mean there's some minor stuff in the rfc that we need to sort out, but.

A

C

Make a list of outstanding questions and then prioritize the ones that block the rfc, then yeah.

B

um I'm just gonna do it in the working group dock here.

A

Sounds good yeah, I assume emily has.

B

The first one I changed suggested.

A

B

Yeah, so the first one is, I think, stack tamil, it's both its name and its uh like structure or if it's its own thing. um So in the rfc um stack, tumble is kind of like an analog to launch tunnel, um and it does. It serves two purposes. One is to define directories and files that are always restored, even if the base image changes.

B

So an example would be the package indices for like apt or whatever.

B

And then the other thing it does is exclude things from the launch or the launch image. So you might have like you know, var cache or whatever um so steven, I'm not married to my proposal, but I think he wants to change some things with it.

C

I think that was one big question. There's also a question about life cycle phases. Is it its own phase or separate phases right.

B

Yeah, there's a whole bunch of facets to that. It's like there's builder. There could be like a builder phase, which does two things.

B

Or there could be like, I don't know what to call it like, like two binaries, like root builder plus builder, like one of the proposals stephen had, was the phase actually executes two binaries, which feels like it's kind of a sucky contract between the platform and the life cycle, but um I mean I guess it's easy to opt out of stack packs. That way.

B

The other was like um extender calls builder. So you still have two binaries. One is like like root builder, but it like it handles the stack packs, but then it actually calls out to another binary, which would be just builder, and so the way you would opt out of stack packs is by not calling the extender builder.

B

I call that that extender builder.

B

C

B

Also something about.

C

Phase specific mix-ins- that was a sticking point and I know steven wanted to sort that out before merging the rfc right.

B

Phase specific mixins.

C

Like the build and run prefixes on the.

A

C

A

Is now another.

C

Rfc form, but I think he cared about answering that question before we started pulling the root builder stuff in.

B

Right, I kind of thought his rfc made that all just work, but maybe I'm wrong.

C

I had some things.

D

C

To change about that, but I could accept it as a compromise that unblocks us and then talk about bigger questions later.

A

uh Do you know if he's coming to this meeting, but just late or if he's just double booked.

C

He said he was coming late and that for reals he was only going to be 15 minutes late.

A

Okay, it's like his 30 minutes late to the leadership meeting.

C

He's now 17 minutes late, so any any minute now.

A

B

uh The other things are some of the finer points, I'll just call it finer points uh of stack packs.

C

There's some questions like what's the working directory, if there's no apps.

B

Yeah, that kind.

C

B

C

Yeah, because I think the rfc was saying that the contract would essentially have all the same parts, but I'm not sure they actually all make sense in this.

D

Context still could be the app directory, it could just be empty right, just give it an empty mouse or something just to keep things simple.

B

Yep I was doing slash just because I well actually, no that's not true, I was gonna say because it felt like that's what would be expected, but maybe like home slash root or something is more like. What's expected, I don't know, I really don't know, can they modify the bomb? Can they add layers? Can they make slices.

B

Can they create.

C

That one's easy and the answer is no, because they can't see the app they can't possibly make slices.

B

Well, I yeah, I mean I guess it would be like. Can they read like we would have to redefine slices like? Could they put each other.

C

In this thing,.

B

Yeah right could they put stuff under user lib into different layers, I'm inclined to say no, but at least.

C

That'd be an easy additive change to make not something we.

A

C

Decide up front right, yeah.

B

So we'll just say: no: can they create process types again very similar to slices like easy to add later? Don't really have a use case yet, but I can totally imagine it on the other.

A

B

Like understanding, if we want to go that direction with these things, I think relates to the question about stack, tommel and launch toml like if we think there's a future where they can create slices process types modify the bomb. Well, that's all that one's a little bit different, but um that that would like lead lead us towards a single tamil file that comes out of the build process, whether it's a stack pack or regular, build pack.

C

So should we dive into the stack tunnel question then.

B

Sure do you have a, I think I know. Maybe you know where stephen stands on it. um What do you think.

C

My hot take is that it should just just use, launch tommle instead of having a separate file, um and it could have its own section, that's specific to this type of bill pack, where it puts things, and we can say some things are not allowed in this context for now, but it make it easy to incorporate other features later.

B

Yeah me too, the only problem is launch, as a name feels too narrowly scoped in that case, and maybe that like why? uh Because some of the things one of the things doesn't have anything to do with launch, and that is the list of restored directories.

B

um So the like the app package indices. Basically, they have nothing to do with launch and you need a place to define those.

C

We now with one of the other rfcs we approved, have a lunch tunnel and a build tunnel. Maybe some of these things blind, launched another one's blowing and build.

B

I forgot about bill domo.

A

That makes sense.

B

But yeah I I just created that, so I could move forward the stack tumble I definitely like or prefer like trying to use, launch tunnel or bill tommel.

C

I also had one more question, which was in buildpack.tamel: we were using privileged bullying to decide whether this was a stack pack or a regular bill pack, and I was wondering if, instead of doing that, we wanted to have something more like. I think the fact that stack packs have to be privileged feels more like a requirement or implementation detail. Then it feels like it's the point of the stackpacks in some ways the point of the stackpacks is that it's making only abi compatible changes.

C

That's the real big material difference from a user's perspective.

C

I was wondering if we wanted to say something like scope: equals stack versus scope, app um to make this distinction, because then later we could have privileged.

C

Still be a boolean if we wanted other types of privileged build packs, we haven't taken this as the key for stack packs, in particular.

C

Like I'm not sure we'll ever do that, but we could make our lives. We could leave that open as an option. Just in case I know, we've discussed it in the.

C

B

Yeah, I don't have a strong feeling. um I I kind of was using privileged as a placeholder until I figured it out and then um everybody just seemed to be okay with it. So I kept using it, but I don't care.

A

I I like the scoping thing that emily suggested um privilege always rubbed me a little bit the wrong way, mostly because of the I guess, the origin where stack packs came from of like we want to probably back out to root, build packs, eventually right as a feature down the line where it is a non-abi compatible change, but that's like a totally separate mechanism and probably want to discourage its use, except for very finite use cases if it ever comes down to that.

C

I feel like it also applies to users like if they want to do something in their bill pack, but they can't because they're not rude they're like I'll, just make it a privilege, build pack, but that's really not it's a whole. It's a slightly different concept than that. This might be more explicit.

A

Yeah and privilege is a I guess like if you we always talk about the darkerisms um privilege is a very specific darkerism thing. um That means a specific thing, so I think that makes sense. Welcome stephen, hey! Sorry,.

D

At least at least, I actually showed up this time, though um the I don't interrupt if you're.

A

Deep in talking about particularly.

B

You're a good good time, I'm plus one on scope.

A

Yeah uh the we put kind of we, we just started to catch you up uh listening kind of the open questions and things we wanted to talk about in the meeting today. um I know, there's a bunch of stuff uh that you probably have opinions and input on. There were.

D

Two really specific things: one was a code or like an architecture thing about how we run the builder and how much we share between root, build packs and non-rebuild packs and the other was mix-ins. If we talk about those two things at some point, then I'd be very happy. I shouldn't.

C

Be both of those questions on our list which we put in the in the docs. We have life cycle phases which I think is getting at the architecture thing you're talking about yep.

A

C

Phase specific makes sense.

A

C

uh Other than scope moving from privilege to scope, the other thing we discussed briefly was stack, tumble versus reusing launch and build tommle with special keys for stackpacks. Do you have an opinion on that.

D

All right! Moving from what to what, with special keys for what.

C

B

So the stack tunnel in my poc, which I think you saw, does two things. It defines the files and directories that will be restored, even if there's a new base image so like app package indices, and then it also defines files and directories that will be excluded from the launch image like var cash or whatever, but that you want to keep uh for performance reasons, um so stack thomas, probably not the right place for those. I just did that, so I can move forward.

B

So we were talking about putting the excludes in launch toml and the restores in build tunnel.

D

Excludes so so I remember there's one caching mechanism where you say this is a chunk of stuff that I don't want to go into the image right like you can exclude some punch image, the launch image, and then you can also pass a cache flag to say also for that stuff.

D

Bring it back on the next build right um which which of those things we just talked about, is that mechanism.

A

B

Wait, I didn't understand the last part, so the excludes has the paths and then cache boolean got. It excludes.

D

As that uh and then then, what's the other thing, that's like the.

B

Stores is the package indices for, like that's only for build time where you don't want a new base image to invalidate your cache.

D

Doesn't doesn't that excludes with the path and the cache achieve the same thing.

B

um I'm not sure we can make that assumption.

D

Because it's like that's how I imagine that's why I thought we were adding a cache to the excludes. So you can say like here, are the apt indices right.

B

Yeah is that I mean is that going to be true, though I mean I'm just not sure. I know enough about the use cases to say that unequivocally, like I definitely seems like there are times where those will be the same.

D

Paths what's the case where it's like. If you wanted to put something in the image and you wanted to restore it or something is the case, that's but then then you'll start on that top of that image. Next time, anyways right, if you can restore it, I thought we just had one one caching mechanism.

D

It was like here's a chunk of in either image here, here's a chunk of stuff that I don't want to make into the final image.

D

You can just exclude it completely because it's temp files or you can make sure that it comes back on the next you know like, and you can make sure that it comes back to the next build optionally, right and so for app packages. You say I don't want this in the image, because you don't want your run image to have the app index and you wanted to come back in the next build. So you do exclude paths equals var, app, something and then cache equals true, and then I think you're golden on both images.

C

In some ways, what we're talking about is kind of like slices right. If you have a diff that your bill pack created, and then you use these paths to slice it, then you have multiple layers and you can think of it as analogous to how we have cache launch and build for normal errors, except the build, doesn't make sense for stack packs because there's no explicit exposing and whether it's launch, true or false, I think, isn't something the build pack decides, because if it was created while running on the run image is launched.

C

True and if it was created while running on the build image, it's launched false. So the only boolean we have left that we care about is cash.

D

That makes sense to me I'm still confused about what the other list is, though, or like I just I just don't know what the other yeah. I didn't see that other thing that was proposed.

B

No, I just I mean the reason. There's two lists is that I wasn't confident that we would be able to capture all all the things with a single list um and it sounds like we can.

C

I think we may need to indicate uh when we store these things in the cache. What.

C

I guess whether it's launch true or build true, um which are exclusive, mutually exclusive in this case, and the build pack wouldn't be actively deciding what to do with those based on that context. But we need to know so. We know which image to restore it to later. Right, like you, basically need to restore on the run.

C

C

Like we can cache build image changes and we can cache run image changes. We need to know the difference between them. Yeah.

D

On the looking through the rfc, where I see the very bottom, you have stack tomlin, build.restores and run.excludes, and I think it looks like you. The idea was to split the runtime version of that with the build time version. But then, then, a while ago there was like, like three or four working group meetings ago we talked about.

D

We actually need the caching on the build time image, also because you're not guaranteed to come back on the previous version of the image at build time, because your upstream base image can change, which I think flattens those into the same api for both images.

D

Maybe that just we, I forgot to leave that as a comment on the thing, and so you didn't uh update it. Does that seem like it's? It's right.

B

Yeah that makes sense.

D

Cool emily, I didn't completely understand the restore thing. I don't know if that covers it, though, or.

C

It depends whether that covers it. Like I, don't know exactly what it'd be nice like see a concrete example of the file written up, so that I know whether, like oh, yes, it's enough information to do exactly what we need to do. You know what I mean.

C

Actually we always get to this level of detail where um we mostly all agree on what we need, but it's hard to get all the way across the finish line without, like typing an example.

A

B

um As I start to flesh this out in the code, there will be some unit tests that help illustrate it. I don't know if that's real enough, but I think they'll be pretty contrived, but maybe they maybe if they, if we replace those with like real directories, that make sense in the context of like apt or whatever.

C

We think because this one feature is uniquely complicated, that the our proof of concept is going to keep feeding back on the rfc for changes. I guess there's like a meta question of how we want to handle that. Do we just want to keep this open for a while and keep working on it and edit it or do we want to get something through and know that very quickly, we'll have another rfc, that's like superseding.

B

Yeah, I think at this point we could probably say stuff goes in launch toml, which will be defined in a separate rfc and then, as I'm working through the poc, we can, uh or I can write the rfc for the launch tunnel changes.

B

Does that and then that we can go into much more detail there than like if we, if we start doing that in the mixins rfc, it feels like that's gonna, get really big.

D

So you're, going to just help me understand: you're going to move um stack tamil to launch tomml is the idea and where does where does launch tunnel is how's? It how's it different than than having stack tumble with an excludes and paths and cache.

B

So I'm not a hundred percent sure but like launch tomorrow, would go in the same place. It always goes the layers directory.

D

But we can't have a layers directory for the stack pack within slash layers, because it's gonna run its root and all that stuff.

B

No, I think we should.

A

C

D

I don't know if we want to do. We want to have some layers and slash layers as root owned and others as no.

B

No, we can't create layers, it can't create layers, but we will pass it that directory to which it can write. Files um like like one of the cases that I was thinking of- and this is because, like one of the build packs I was using did this, it was just like setting up some files and directories in the in the layers directory because that's just what it did um and then my builder just ignored it, because it was a stack pack and I feel, like that's, actually a real case.

B

We should consider I'm not married to one behavior or the other, but I I kind of favor like yeah, just let it write stuff in there and ignore it like, if, like you, can imagine if you're using like lib build pack and it initializes some file or thing like, even if it's just like it's a sandbox. Just let it do that. Don't air.

D

Out so you just mean by a layers directory, you just mean there's a directory that it's dedicated to the build package, buildpack used to construe it to store configuration and communicate that to the life cycle. That makes a lot of sense. I think the calling.

A

Of the leader's.

B

Directing word sure.

A

Sure- and um that is what we call.

C

It, though, in the biblical api right.

A

It's the build.

C

Packs layers directory- that's the one, that's like slash layers and then build pack id and then underneath that there's the layers layers directory.

B

Yeah, well, I think I actually think layers directory is a misnomer, like the layers directory is actually well. It's like the layers directories, which are probably in the like, build packs, sandbox directory or whatever we call it like. That's where it launch tunnel goes. That is it's not really a layers.

C

You know there are three different things we call layers directory. One of them is slash.

D

C

Then the build pack api, we call slash layers, slash, build pack id the layers directory and then we talk about a layer. We call slash layers, build pack, id layer, name, the layer directory.

D

I think we're we're in agreement. There should be a directory dedicated to the stack pack. The stackpack can use to communicate configuration to the platform it doesn't. It's not used to preserve bits that go into the image and into the the final image.

C

I think it should live under slash layers, flash fill pack id. I think it's just easier because we have a single volume and we never just iterate.

A

C

That blindly, we always use the group to figure out what we want to read out of there, and that would be similar. We just use the stack group to figure out what we want to read out of there. I.

D

Don't have a strong opinion we can yeah, we can move on. I have issues about accidentally preserving that directory in the future, but like it's, it's I'm happy to compromise that.

B

D

B

About probably, I think we should probably rename it. I think it should be the same directory, but I don't think it should be called layers.

D

Each of the build packs layers directories like the the layer for each the directory for each build pack, for which the build pack puts layers in contains directories that could turn into layers. The only thing that's in that top level, slash layers, the only thing that's not layers is some configuration files that sit around layers. So if we didn't want to start, oh there's.

C

Put a bunch of layers of other random files in there too, because it can.

D

Just stop doing that, but then we need.

A

Another place to put.

C

And we don't want another volume in some ways. Layers should have been a subspace underneath the what should have been the workspace and then the app is the app I could.

A

See that let's yeah.

B

Just for time, I think we're enjoying it when it's left.

A

B

uh Let's talk about the life cycle phases, because I think this is important to my poc at least moving it forward. um The way I the way I have it structured now is that there's one builder one builder binary and also the like sort of the library part of it, which is like builder.go at the top level and they're all the same thing. There's the same entry point.

B

um It just accepts a group tommle and a stack group tunnel that the binary does, and then it runs the library part of the builder twice once for the the stack packs and then once for the regular, build packs, I'm completely open to changing this, I'm just trying to find you know ways to not copy and paste a bunch of code, including tests.

B

And but I think so, steven had proposed splitting it out into two builders, a root builder and a builder, but they would have to run in the same container in the same phase. So that would require the platform to know that it has to run both of them, which is a little bit weird as a contract. But.

A

D

It's not not exactly that. um Can I give a little bit of motivation and then talk about what whatever.

C

Some rules for the discussion or a principle. I would like to propose.

A

C

Which is, I think, some of these conversations have been driven by not wanting to repeat a lot of code, but I think there are many ways to avoid repeating a lot of code and I'd like to start just by framing the conversation on what interface we want. And then we can talk about how to not repeat code while doing that.

D

I agree completely: um I'm separate not related to repeating code, I'm I'm worried about having the builder logic right, the the thing that runs unprivileged and the container runs through normal, build packs like as we extend that I worry about applying the same logic in the on the run image during build time when you're extending the run image and just running stack, packs or like in all these other cases. I feel like it's a it. It's sort of built to do this. You know running normal, build packs in a row thing.

D

uh You know, you know against that contract right uh and it's it's nice that you can. You know always run it as an unprivileged user and expect it to you know, write all the files with the right permissions. um You know all that, so what I propose is uh have it another binary. That's the stackpack binary that runs privileged right.

D

That, um in the during the build phase, runs the stackpacks with the stackpack contract first, and then it actually executes the unprivileged builder as a normal user and does the rest of the build packs using the normal builder logic. If a platform doesn't want to support, stack packs and wants to do a maximally, secure configuration or the build image always runs unprivileged like kpac.

D

It could just execute the builder and not execute the root builder, but if there are stack packs, the platform can execute the route builder, which executes through playbacks and then executes the builder. Then, on the run image you could have either a flag to that privileged binary or a different binary. That does the extension of the run image running the stack packs on the run image and that's the without talking about the code. That's the high level kind of like architecture that I think would feels most secure and keeps the best separation between.

C

D

C

Know why we have to have two binaries to achieve all those things you just laid out. Stephen.

B

Like if yeah, I was gonna, say, that's like what I'm doing right now is exactly what you described in a single binary like, instead of call like, instead of exiting or calling out from one binary to another. It's just happening in the same binary where it switches users.

C

It drops the privileges just like we do when we run creator, so you could either run builder not as root, in which case, if you passed it stack, build packs you'd, be like no sorry. I can't do that or you run it as root and it does the stack bill, packs and drops privileges and then does its normal thing.

D

There are a couple things that make me worried about that. One is the this is when unf untrusted code executes right.

D

The way we've dropped, we've kind of figured out how to drop privileges in a go binary with a degree of reliability, but it's very easy to like in the first couple iterations, for instance, not not drop privileges in all threads and have stuff that's secretly still running as root right. We.

C

D

C

D

But it's not a simple thing to do right. um The.

D

It feels I'm not super strongly opinionated if there's still a way to run this one builder right as a normal user and then have it, skip the stack packs and then run the normal builder logic. I think I'm okay with that, but it it feels simpler if, in the case, where you're not using stack packs, the platform can execute something. That's just the normal, build pack runner and not have to worry about what that logic looks like.

C

I feel like in the case where you're not using stack packs. That is exactly what you would do. You'd execute the normal builder as a normal user and you get the same behavior you get now absolutely nothing changes. It's.

B

Just like yeah, the only.

C

Thing that changes the stack pack example.

B

Yeah, like the builder binary that I have in my prototype, it only like if, if you pass it, the stack group tamil, it checks to make sure that it's privileged and if you don't it, doesn't or actually I think it's the inverse. It's like it checks to make sure it's not privileged.

B

If you don't give the stack uh group tommle so like you can run that that same binary with or without privileges,.

B

I feel like the separation you're, describing stephen feels like like a uh what do you call that not like a security thing but like a trust thing, and that, like this, ensures that we will not accidentally run something as root that you didn't mean to, but I feel like.

B

We could still do that by accident like if with even if we have two separate binaries.

D

Yeah, it's definitely uh terence has a comment too.

D

um I appreciate the support.

B

I don't feel strongly either.

D

Way, um I don't feel super strongly about. Well, I think I I lean towards liking the process, separation right and keeping keeping the whole builder binary. As you know, strictly you know, even just for internal factoring reasons. You know, sort of strictly separate from the thing that runs stack, packs, so platforms can choose to. You know, never have to deal with that code path if they want to, um but I could like I'm not going to block the pr on it as long as the internal factoring of the builder itself.

D

Is you know similar enough to that thing? If we feel strongly that we can, we can really guarantee that the life cycle can always adequately drop privileges on all threads and then we're not gonna. You know, and and also that this is not gonna cause a problem on windows.

D

um What's the reason we have to in both scenarios it's sort of assumed that it's a single it's either a process and a child process? Why can't it be a separate phase? I guess I'm missing where the.

B

Oh, the yeah, the separate phase would require a separate container and then you would lose. There would be no way to carry forward the changes that you made to the uh to the image during the stackpacks phase. Unless you also gave privileges to the next phase, so you'd be you'd, be in trouble there. You know, you know what I'm saying like. If you.

A

B

Yeah, if you wrote something to like user bin, you would also need to write that in the unprivileged builder phase. But you can't, because you don't have privileges.

D

We're trying to avoid dynamically writing an image and loading the image in the next stage.

A

D

That's what I want to see.

C

D

Don't like having a.

C

Single binary entry point, the part where I care about the separation is it the last time I looked at the proof of concept is running a single build method that could do either type of build pack, and I actually want those things to be separated at the library level, even though they have a same yeah.

B

C

B

That was more like it was just really easy to do it. That way, because there's like two lines, different and or maybe maybe a little bit more than that, but like how it loops over the build packs and how it sets up the cmd to execute. And all that was exactly the same. So yeah like, like you, said, there's other ways to accomplish what I don't want is.

A

C

Someone put privilege true in their bill pack tommle and then put it in a group right that we would get any of this behavior. I feel like if it's in a group. It should never go through the part of the code path that does yeah. I agree.

D

I'm yeah I'm worried about, like we put some logic in either the privilege builder or the unprivileged builder, and just just just purely as like a as a potential for introducing bugs it. It applies in you know one case in other case without us, without sort of clean separation between the two, but as long as there's, um you know really sufficient separation at the library level, then I'm not going to block if emily.

D

If you feel strongly that it really should it's too complicated to have an interface where you know the platform decides what binary to invoke, um then I I I think I'll. I'm I'm definitely willing to compromise on it being one binary, but um if you have information on how we're we're like, I know we're talking, this is easier for the privilege.

C

Authors to have one binary, especially because most of the inputs are the same, and it just makes sense on some level.

B

Yeah and I feel like if we can't do this or we don't have confidence that we can do this correctly, we shouldn't have creator at all if like, if we can do it for creator, we should be able to do it for this. Well,.

D

With creator, though, we're saying that you, you trust the code, I I'm my security worries around this come from. I expect the api to be used with build pack code that is like completely untrusted and creator is like not safe to use with completely untrusted. So.

B

So, let's make builder exactly like creator where you get you can do it either way like there's a builder, you know like a top-level builder and that's sort of the one. That is the default that you go to and it just does everything, but we can generate two other binaries that are like root builder and or like privilege, builder and unprivileged builder, or something like that.

A

B

Then, like you, just ignore those unless you're like power user, who wants to be like nope, not running stack, packs.

D

Make it have have always, essentially you could you could make one that that does this runs this yeah? I don't, I think, that's I still, although not very strongly. You know what like the separate binding areas that do one feels very unixy. I guess I have separate binaries that just do exactly one thing, um but that, if you know it would make me feel better if there was a binary that could just be the builder and then a binary that did the root thing and then without having to invoke the other binary.

D

It could also execute the same logic as in that builder, that that feels better than not having that other binary that doesn't have the other stuff as an option at all. Right, um but again, not not super strongly opinionated at this point, as long as there's really clean separation at the library level,.

C

We just be really explicit about what the security concerns are when you're talking about running untrusted bill pack code as root like for other phases, that we have trusted and untrusted configurations, for that makes a lot of sense to me, because there are sensitive things like access to the docker demon socket or the registry credentials that we don't want to provide in an untrusted situation or to like an untrusted life cycle.

C

But what is the harm sort of of running on trusted build pack code as root when it's not like a privileged route? It's just a regular route.

D

uh I think there are some usability concerns right with like now. You can write everywhere, but if you ignore, if you ignore that and just focus on the security thing, um there are some in container exploits that are easier if your local root right, um especially if you don't, if you're, not mapping, I mean, if you're mapping in container route to another user, then it's it's.

D

You know not so bad, but um there are capabilities that, if you accidentally grant to the container, you can't do much with if you're a normal user, but you can do you know if you have net admin right, you can do some pretty crazy things with this route. It's it's. Definitely if every other, you know thing is set up perfectly great right: it's not a problem and many docker file builds.

D

You know conoco, I think, executes his root right in the container, so we're definitely being very paranoid about it, um but the you know I given that this is sort of a thing we're designing for really large platforms to run lots of untrusted code. Where you know every third build pack is a cryptocurrency miner right. um The I want to be very careful about not having any privileges in places where we don't need them. If that makes sense,.

D

But again, like I agree with the single binary, I don't have a strong argument against the single buyer name of the drops privileges as long as we're sure that it dropped privileges.

B

I'm going to keep doing that in the proof of concept, and then I think we can dig into it in the code. You know like take a look at like once like. I definitely need to make the library separation um and then take a look. We'll take a look at what the binary looks like I'll. I might actually push up a layer from where it is so that it is very much like creator and then we'll just see if that you know meets your expectations or your makes. You happy.

D

um The I had a question I.

A

Just forgot oh well,.

D

Never mind um that sounds good. I I think I uh I'm really interested to see what it looks like when they're, when it's sort of they have a separate root, build pack builder. You know in the library and then you know you have a binary that invokes that separately and then then drops.

C

Privileges, so the next seven minutes can we find what makes sense.

D

I'd like to, but I don't want to move on too quickly if joe, I want to.

B

Know I think this is perfect. Yeah.

A

D

A

D

So I'm willing to compromise on this. I don't want to lose any functionality we have already as part of the rfc, like my intent here, isn't completely refactor mix-ins. It's um get it to a point where you can use it with stack packs with with application, make sense.

C

I wanna propose that run. Colon makes a name.

C

The delta of changes on the run image should be exactly the same as if just mix and name was on the run image. I think we use it to sometimes mean a slightly different set of changes that don't correspond to a paired set on build, or something like that. I feel like. If you want a slightly different set of changes, you should just have to have a different name for it and therefore run colon name should describe the same thing. Name does in the context of a single image.

D

um Okay, how about I say done: let's do that. I think it makes the mixing contract a lot less powerful, but I don't have good use cases for those the first first like threat in rfc. um It.

C

D

C

You can always define a different set of things. You just have to give it a different name, which I think makes things easier to understand.

D

There are some weird cases like I'm going to give you an example, but I'm not pushing back on that change like. Let's just do that, because I think in the case of ubuntu packages, it's not going to matter that much because a package name literally means install this package, but there are some cases like you know you mentioned. Well, if you want the dev package, you just get the dev package.

D

If you want the not dev package, you get the not that package, but the dev package actually installs the not dev package right, and so you having the dev packages build only and having the regular packages not build. Only you know, you'd see those on the images and the regular package would imply the contract across the stack. So it's not.

D

There are some cases where, like it's convenient to be able to say you know this mix and represents the run portion of this, with no expectation that it's not in the build portion, but in the context of a stack pack receiving the mix in names, which is the only place this matters, it's it's not going to matter. If we're going to package this because in the end stackpack just needs to it's already on that image, it just needs to install the package because we don't have a wide use of mix-ins outside of packages.

D

I am okay. Relaxing that restriction in the mix and contract the the next.

C

What you're saying about certain things implying requirements of the other image is still true, and I'm not arguing with that. I just don't think that the name should describe a different set of changes within a the run context or a build con or a build context. It's like the difference. It can mean different things in build than it means in run, but I think run colon on a run image shouldn't mean something different than the name on a run image.

D

Okay, and that means that stackpacks receive unprefixed uh uh information and okay, I will make the change to the rfc this week. um The next discussion is what I was more worried about, or the thread under that we had, which is in the stack definition. We still need the prefixes in order to imply the contract between the stacks, and I thought you were saying you wanted to drop that and that's that's the thing I was most worried about, because that'd be a really big breaking change.

C

On the fence about dropping that, I feel like yes, if a certain package exists on the build image, there are things that can be built on that image that then put requirements in order for them to run successfully. On the run image like I agree with you that there is a contract between the um between the images when you have a particular mix in, I guess my question was: does that contract matter? If none of, if the mix-in isn't used by any build packs like?

C

Can you let the fact that a build pack needs a mix in and it hasn't prefixed it describe all parts of the contract.

D

um No because you don't know what the application is going to use, the make sense that the build pack specifies are just mix-ins that the build pack needs.

D

If you have a ruby build pack, you might require libyaml, because ruby needs libyaml, but if you're building a ruby, app and needs image magic and you build up with image magic dev on the build image and image magic isn't on the run image then you're in for a completely great looking build experience that doesn't fail to launch with a very surprise that fails to launch a very surprising error message if you want to.

D

If, as a stack author, you want to say that all your mix-ins have build colon on the build image and run colon on the run image.

D

You can implement exactly the api you want, which is it just doesn't matter at all right um with the and this the first change we talked about will allow that where it wasn't allowed before, because the mixins didn't I mean the rfc in general will allow that, because having build colon package name and run colon package name on, the different images now satisfies just package name coming from the build pack, and so I think if we, if we just relax that first restriction you talked about, then we have all the functionality and you can still apply a contract between the images if you want to, but it's totally optional.

C

In the case you're talking about where an app needs image magic, there's a mixing requirement that nobody has expressed right. So you could try to build that on a build image that doesn't have image magic and you won't find out until it fails. But what you're saying is, if it succeeds, it should keep succeeding.

D

The user has the user has decided to use that mix in by selecting a stack image that has it um or or eventually you could say, the user could just put something in project tamil that gets sent to the mixon's requirement and then the mixing gets installed automatically right. But I don't think that's always the.

C

Case yeah, I want to think about that morrow, but I don't need to push it now, because it is a big change. It seems like it's got risky edge, edges.

D

Are you, are you, okay,.

C

Yes, I can see, I can see okay.

D

Just just getting the rfc.

C

Through or the application makes sense.

D

Done we'll take your thing for.

C

The first time for the second thing, yeah.

A

D

Awesome perfect, um so then joe, I think we're unblocked on the mixer interface and stackpacks can just receive unstaged specified. Unstaged specified is kind of key here for your your rfc package names and just wrap them straight to appgate install, and it should should be really clean.

B

Yeah that sounds great, um I'm gonna have to drop uh the only two things that we didn't get to. Actually, I think, are, can stack packs right to the bomb and what's the working directory- and I don't, I think they should be able to modify the bomb, but uh I don't care about the working directory.

D

um Seems, okay with I have to think about that because of the replay stuff is weird, but there should be some way to write metadata for sure.

B

Okay, yeah think about it: it's not a blocker, so yeah.

C

I feel like for now we could do a naive, they add to the bomb, but I.

A

C

If someday, we actually want stacks to come with bombs, and then this adds to the stack bomb. You know.

D

The building should add to the report. The runway should add to the bomb yeah.

C

Yeah they're both bombs in the yeah.

D

But it's not a bill of materials, it's a bill of tools that were used to generate the. I don't know it's a.

C

Bill of materials for enough foreign.

A

B

Okay, I don't, I don't think the most secure teams would differentiate um all right, so I'll update the rc with this uh and then I'll keep working on the poc cool.

D

B

A

Let me know if you.

D

Help or tag me on github.

A

Cool cool, bye.