Paketo Buildpacks Working Group, 15 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Working Group: 2020-09-15

Description

* NodeJS family no longer using tini
* Paketo Stack Overflow tag
* Builder Versioning: https://github.com/paketo-buildpacks/builder/pull/36
* Docs, common concepts

A

A

B

Looks like a fun place to be.

A

Yeah, it's nice pretty breezy, though.

B

That's why wearing a sweatshirt.

A

A

Also, it's cold in new york, so.

B

It's true: it's like 50s this morning, which is crazy.

A

It's messed up.

B

How did it drop so.

B

B

C

uh As a reminder, please sign in on the agenda.

C

C

Foreign get started in two.

C

C

All right are we waiting for anybody else.

C

Probably good. uh First thing in the enchanda is review outstanding rfcs. I will share my screen.

C

Everybody see that.

C

uh So we have a draft here.

C

Anything does this need any kind of attention.

D

I think that tim has uh still looking into a couple of just like questions that uh are in the original draft and trying to get those resolved um exactly where he is right. Now, on this.

C

And the next one is dependency server. Rfc looks like uh about two weeks ago. Then you pointed out that there was a uh remove the template by accident.

B

Yeah, we're still waiting for mark to come back from vacation to just fix this up, since it's on a fork in his account.

C

Got it wait for wait till he gets back and then rfc for dependency, mappings.

E

I think this one's waiting on your approval, stephen.

C

And this was just how what dependencies look like in the thing yep, I I just haven't, hit the button. Yet I like that one okay.

B

C

And anything else, rfc wise are there any sorry.

D

God, actually a quick question: uh I'm putting together an rfc about a workflow to have a staging site for the picato website and as well as a production site, so that we can shop around updates to it in a more effective way than telling people to run a series of cli commands and then host it on their machine.

D

um Where would where would you like me to put that rfc? Would you like me to just localize that into the piccata website um sort of area, or do you want me to propose that to this place as a whole? The only thing that we'd have to do is we have to set up some cnames for the website, so.

C

We have like a learning team or like a sub team, that's around ensuring our documentation is this together. I forget. If we we have one.

E

Of the clogging, because I.

C

Have a content team that probably be good rfc for that sub team: okay, cool I'll, just chuck it there then I just wanted to know, but then does that make sense to you yeah yeah. That sounds fine to me. Cool sounds good and anything else before we move on to- uh or I guess if there are other rfcs people want to chat about the individual language group repos. Please add that to the agenda doc this week and we'll start at the next item, which is node.js family no longer using teeny.

D

Yeah I put this in here because steven you and I had a quick chat about this and I figured that we should have a chat about it, not between just you and us. uh You, you and myself.

D

uh But um so we tried introducing teeny to allow us to use npm start. We have discovered that teeny does not actually necessarily always work to kill all the processes that are spun up by npm.

D

Npm is very resilient in its fact that it wants to let processes live, um so uh we have decided to stop using teeny. I think we'll keep the bill pack around. We might actually move it into community. I haven't quite decided yet, but um in the meantime we are replicating what npm and yarnstart do by investigating the package.json and then running pre-start and start commands, and if those don't exist, we just do uh node.

D

Server.Js is our start command, because if you just execute that looks like everything's cleaned up for you pretty nicely, um but steven you mentioned that we might not want to do that.

C

It was more, um I guess, a philosophical thing so pulling the start command out of package.json from the location that npm start would execute it at and then like attempting to recreate the behavior of npm start. um It feels like we're it's a lot of magic that the user might not expect that, like the I'm not aware of any other tooling that looks in you, know, package.json and tries to pull commands out of it and execute them separately. Right, it's like that's, that's the configuration for npm start and users.

C

Expectations would be that if you run npm start it, it starts that you know uh runs the processes in there in order defined by npm right. um You know, I wonder if it'd be better, if we do something simpler and just let the user specify a start command, you know somewhere outside of that.

F

Do we have an idea of how, like I, I generally believe that or agree with stephen on that that, like trying to replicate something that another tool does is almost certainly going to get us into trouble? Do we have an idea of how often, like npm, has changed what it does for these sort of alias commands.

C

I think that's part of why I'm worried about it is, is they've changed some of the pre-start and post art logic over time and like whether things get executed, shell or not, um and so like? I, don't I'm not suggesting that I have an idea for a better interface, but it seems like if they're saying npm start isn't. One thing for us you mentioned before is um npm start isn't really intended for production, and so, if you have a command in there, that's like the command for starting your app.

C

That's for use with npm start it's not intended for production. Is it really the best command for us to use to run the image in the end? um I'm not I'm not saying don't do it, I'm just saying uh you know it seems like. Maybe we could get away with doing something simpler that doesn't require, like you know, creating this abstraction. On top of the uh you know, our alternate way of you know, reading uh package.json and executing it yeah.

F

Like I think, I think, being conservative here is probably the way to go at least initially right like if there is a better command for us to run, one that is more manageable, that doesn't involve us replicating the interior logic of npm. Let's do that and let people complain and sort of build on top of that once we we see the complaints and we think we have a safe way to do it, but being predictable about this and not replicating code. That's out of our control, I think, is the responsible way to attack it.

D

So I'm not quite sure what so are we saying that we should just go ahead and run npm start and wait for people to complain that we're running a command that does whatever or should we just run, none of the pre-start logic and just run node server.js, as if there are no commands in package.json.

C

I'd say either defaults at node server.js or have have another buildpack specific way of specifying the start. Command is as terrible as that seems and that we're creating another interface, like the I think, they'll just set up a proc file if they really want it. Yeah.

E

Well, yeah yeah. I.

C

E

Why we haven't sort of included the proc file build pack in all of the language groups, anyways.

G

So like proc file is great. It allows you to override the start command, but, as you started, to move towards being way more conservative in what we actually include in the launch image, procfile doesn't have any controls. So, let's you say like given the fact that you've said nodeserver.js as the web process in your proc file. Therefore we know we have to ensure that node is available during launch right. That's the other kind of reasoning for having these start command build packs.

G

Is that they're able to actually interrogate the contents of the app to say like? I know that this for a fact needs to be in the launch image instead of just aggressively always saying include launch, you know include node and launch.

C

Got it so the problem is that we're using the um start command in package.json as a key for knowing that this is supposed to be a runnable app, as opposed to some static assets that you want to build and then like host with them with nginx yeah. So is there another indication you can use in package.json? That's like that suggests. This is a a server side app as opposed to a like. You know some static assets. You want to.

G

Build uh not that I know of I mean we can continue to look into it. We could also build our own interface for doing this. We just get into a position of like currently, the behavior is like, if you have in your package, json script start field, which lots of folks are doing just as a convention already, then this just works um versus we're.

G

Gonna have to build our own thing, and it's like okay, you have to add this new file that you're not familiar with, add some more fields to it, just to get your app to like build for the very first time.

F

What's the prior art out in the world like what did google's build, packs and heroku's build packs do around this.

G

I haven't looked at that. We should.

C

It seems like it probably don't need to make a decision here, but maybe that's a good some good next steps.

C

um I I think I'm I'm actually a little bit less concerned about this particular decision, if, like a lot of other, build packs, read that command as a shorthand way of doing things, it's more the in general for the past generation of build packs.

C

We had focused on like creating an abstraction on top of language, tooling right, because it's you know built into this integrated platform right and a lot of the features we have were convenience feed like like kind of abstractions around existing language tooling, and I think that made things very hard to understand for end users who are used to their language tooling, because they, it wasn't very transparent and so, like I'm not saying we aren't already moving in this direction, but it might like for the next generation of build tags.

C

I wonder if we could look at it more. Like you know. The value we add is, is, you know, supplements the existing language tooling, instead of tries to build up on top of it, and this kind of struck me as like: oh building, an abstraction on top of you know, npm start now. If that makes sense, um I don't know if that's helpful or the way other people have been thinking about it.

D

Google's uh build packs, run npm start at least for npm.

G

Interesting so then they don't handle graceful shutdown any better than we do today.

G

That is the implication.

F

A

Yeah, sorry has anyone complained about this, grateful shutdown thing or is it.

G

Yeah, so we had a, I think it was a pa cora who came in the channel and was indicating that we don't support graceful shutdown in our containers on node. That's how we first became aware of the issue.

F

Yeah since and she she'd be the person, she drove a lot of the graceful shutdown features on the spring boot side as well. To make sure that that all looks good.

C

It's definitely been a problem in cloud foundry. Before too, like we even had a diego issue once where we weren't shutting everything down correctly and messed up a lot of databases. So it's definitely an important thing, but the it's hard to get right for sure um seems like next steps are maybe do a little more.

F

Research, um so the fact that teeny had one job and doesn't appear to do that job, particularly well, uh is there like no thought that we should be trying to improve teeny to be better and compensate for the stuff that npm does.

C

I doubt it's a teeny problem. It's probably just npm is bad at launching processes that are owned by its p group or whatever.

C

Might be worth looking into that too.

C

um Forrest, you mentioned some documentation. That said you shouldn't use, npm start anyways for production. So that's pretty good evidence.

D

That we, you know so basically, basically what everything that they've said is like yeah. You can use npm for all this stuff and npm start is great as a developer tool, but like their official messaging, is like who's.

F

D

Npm as far as okay is that, if you want this graceful shutdown process management thing to happen, then you should run the node commands yourself. At least that's as far as I can find, when I you know sift through github documents of them saying not going to fix, so I wouldn't as much as like. Maybe we sort of just surfaced that um and keep using npm start, because I think that people just use npm start anyway.

F

Yeah, but I mean like so the the decrease in usability. I think we we should be sensitive to that. But, like one of the core selling points of build packs, is we codify best practices right? We go to our partners and we say: hey. You know how to run new relic best.

F

You tell you know you put it in a build pack and everybody will run it with the way you think it is, and so there's an avenue for saying: yeah, it's not as convenient, but the node community says you should do this thing and we are codifying that as a best practice.

F

So like there's, there is some defensibility in that that bad, ux or less optimal.

F

D

I think that we've got like we're good we've gone with this decision. To do this. For now we can move to something else in the future, we're just trying to unblock ourselves and move forward. Yeah.

C

um Maybe see what haruka does, but then, if, if it's not anything particularly clever, you know it doesn't doesn't solve the problem, then just just use the command like we're doing from package. Json does heroku have any actual.

D

Just cloud native native build packs, or do they just have shimmed v2a built pegs.

F

uh But I'm pretty sure well they're, making their build packs support both which is shim like, but not necessarily a shin. That was last, I heard, but I don't think it even needs to particularly be a cloud-native build pack like any build pack ever written against. Node.Js doesn't use npm start right because it's if it does it's, not compensating for this. Regardless of what platform it was running on yeah.

D

Yeah absolutely take a look.

C

Cool, so that's not interesting.

G

It looks like heroku just calls: npm start yeah in their.

C

G

In their node build pack.

F

Some days, it's easy to forget like we fight internally, about how good things are, but we are so far ahead of like these kinds of really fine grain enterprise details than anybody else's build packs like we are very good at this.

C

Definitely a selling point.

C

For um all right, next to the list is uh there's a potato tag on stack overflow.

D

Yeah I just found out about this point this this morning. I just thought everyone should know, there's a tag for picado and check it out.

F

Do we want to hook up all of the plumbing to have like to take ownership of it and manage it and get notifications for it like into like a slack channel or something that'd, be cool.

C

Ben, are you volunteering for that.

F

uh I'll I'll report back in a couple of weeks, since I know the spring team has a huge automation flow around it, they can at least tell me what's possible.

C

uh Who's, who are the maintainers on the content team.

D

uh It's like myself, kasha. uh I think a couple people on the cff.

C

Maybe uh you and kasha can help support ben uh in getting a stack overflow stuff set up.

F

Yeah, like in theory like this, is the kind of stuff that needs to go back and like individual dev teams will eventually need to have visibility into this and be watching it as they come in right, like somebody's gonna, ask for node thing or a java thing, and so it's not simply the documentation team per se, um but as a place to get started. I think that's good just meant for setting it up. Yeah.

D

And ryan, are you, are you still our slack admin as well.

G

Yes, I've uh started and tried to restart the conversation of handing over the slack instance to the cff a couple times. uh I will I'll poke the bear again.

C

You got it, you can get him to pay for it. When you do too, that might be a harder ask.

G

Yeah yeah, we we want team aliases and that's like a pay for feature. So it's definitely on my list.

C

Cncf, wouldn't let us do it for uh cloud data build packs, so you know apparently.

E

It's expensive.

C

um All right next thing in the list is builder versioning, rfc.

A

um Yeah, so I guess I'll share my screen really quickly.

A

So can y'all see um yeah, so this is. um We have filed this issue that we talked about. I think in the last meeting, which is that um we should um every like there should be a checked in building a tamil for every um release version of the builder so that um you could easily just look at it and like recreate it or add to it or whatever, and so we wrote this rfc um that is kind of just like a proposed implementation um for what we want to do.

A

So um I think what we decided we end up wanting to do is to um like uh do the entire thing and get them actions, because we didn't want to have to involve concourse in this, and um we also didn't want to have to uh force um like member, build packs that, like build backs that are members of the builder to have to implement um github actions logic on their side.

A

So, instead of having those build, packs send dispatches. What we're gonna do is um on a schedule we're going to try to grab the latest tag of each build pack, that's in the builder and then attempt to create a new buildpack.builder.toml and then, if there's no diff, then just throw it away. But if there is a diff, then um commit it basically so um yeah! That's that's! Basically what we thought and we thought that would be good, because it's the same thing. We have to do for the core depth dependency server anyways.

A

So we figured that we can share some logic. There.

F

Yeah, um the one other thing that uh I'd like to add to this. I I'll put it in an actual review comment as well. Can you mention that you should do the same thing with life cycles, build and run images all three of those uh yeah.

A

I don't know if we're taking those over right now. Yet, though,.

F

But you can watch for them to change right. Builder. Tommel has those things. Oh.

A

Yeah yeah yeah for sure yeah. It would be anything that's inside of the builder.

C

The run image is just a reference to the latest one though so you don't need to watch for that.

F

You can you can, I think we should actively not reference latest. We should reference individually versioned in.

E

The buildings of the run.

F

In the builder.

E

B

E

Latest thoughts, if a new one came out and you rebased you get it this.

C

The the reference is only so that the the latest tag, the mutable latest tag, gets passed to the image that gets generated so that when you run rebase on it, it can find the most update when it's not a reference. That's intended to be like this exact version. It's it's it's strictly intended to be, so you always get the latest one. When you do a uh rebuild yeah, then the build one at the very least yeah it might be.

C

If we wanted to capture the corresponding run image to the build image release in the builder metadata. Also, we don't. We don't need it for anything right now, but would be supportive of doing that as to be able to have that extra metadata, but it.

F

Seems like another rfc yeah, I mean to some extent it doesn't matter right, like the run. Image is never used by the builder itself. It's only the output at the end, so um it only matters when it's actually used so okay. I I step back from that uh life cycle and uh build image should be hard version.

A

Okay, well, I think just yeah, I mean just waiting on approval, I think from somebody, but I think.

F

Yeah I'll go do that.

B

Are you gonna when you pull for the latest, build pack and stack and everything are you gonna pull off the images on docker hub.

A

Yeah we're just gonna ask docker hub, like what's the latest version and then uh go based on that.

B

C

Sense, um so this is an rfc to the builder repo.

A

C

Right and am I or who, what sub team does that fall under and who approves those.

A

We put you sorry, we put um you ben and forrest as the reviewers, I'm not sure which sub team it would have been.

D

I don't know that we have a sub team that owns core depths currently owns the idea of builders, but we're in the process of taking over that responsibility within uh you know our our sort of buildpacks team, yeah.

A

I I guess, that's implicitly part of this, this rfc as well that we will we'll take over the responsibility for for the build for the builders.

C

So it seems like we need to create another sub team, because the core depth sub team right now isn't uh isn't going to maintain builders going forward. It's going to be people who deal with build pack logic more. um Should we have an rfc for that or something yeah.

A

Probably like, like a builder sub team uh yeah, we could we could do that. I think right.

C

Is there like a? Is there an appropriate sub team that exists already like tooling, or I don't know if anything match as well, it could be lumped under tooling right. There is a tooling team github.

D

Actions and underlying library, or it could be.

C

um It just I I don't know if every every rfc towards the builder needs to get approved by you know, governance level. No, no.

D

This is, this is the intermediate while we have well, we don't have everything to set up for the sub team itself.

C

Whatever you think, though, for I don't have a strong opinion about where it goes, I'm just saying we should. We should figure out something for it, eventually, yep cool for sure.

A

That's all any questions or comments or anything.

C

Cool has I lost the uh dock. Is that the last thing on the list today, no uh docs common concepts.

E

This is one I put I've been working on the java bill pack documentation for the last couple days um and found myself wanting to explain concepts that aren't really particular to the java build pack like what is a binding um stuff like that that I was thinking we should be pulling into a place where we could reference from any build pad, although I'm not sure if all of them like right now, I think it's, maybe only the java build packs that use bindings, but it is a concept that could be used by other build packs in the future.

E

um This is one concrete example of like a couple of a category of questions that I have and sort of my. What I wanted from this was to find someone who's working on the non-traffic build packs. You might want to collaborate with me and look at what I've put in the java stuff and talk about how we could pull that into a higher level page in the docs.

D

You know I'd be more than happy to take a look at some of that stuff with you.

E

Maybe we can schedule some time.

D

E

Let's sync up after this.

C

There's like a content, sub team thing.

E

Yeah I've gotten in the spiral of like way overdoing the docs here, but maybe like under doing them on some level I got. I got.

E

C

B

C

C

All right, we'll call it early thanks.

C