Cloud Native Buildpacks Working Group, 30 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Working Group: 2020-07-30

Description

* Project Descriptor Schema: https://github.com/buildpacks/rfcs/pull/103
* CA Certs
* Shell-free profile.d: https://github.com/buildpacks/rfcs/pull/104

A

Are we expecting uh steven or ben.

B

I know ben doesn't come on thursdays anymore, but I think we should be expecting.

A

Stephen, no, he just said he can't make the first half so. Oh.

B

A

Yeah we can go ahead and start. Let me pull up the dock. Make sure you sign in.

A

I'll do that, uh do you have any new faces, anyone who hasn't been to the meeting before or would like to introduce themselves that doesn't look very good.

A

All right, that's it call it a meeting.

A

uh Josh, do you want to talk about your uh schema proposal yeah? I can that'd be great.

C

Let me get it up.

D

D

So this is uh 103 is that right.

E

D

So yeah just kind of.

E

A bit of background I put this together uh because we kind of did on the heroku side we're kind of extending the project descriptor to include more platform specific functionality, mostly by kind of like landing it in keys that we're not using um when we developed that we ended up making a json schema for the like merged version and thought it might be useful uh for just the general project. Descriptor.

E

Some of the some of the nice bits that come out of this, as you get um slightly, I guess right now, it's kind of like we have a bunch of rules about what a project descriptor is and um they're all hard-coded, meaning, like you know, we we have to like explicitly write tests and explicitly like make sure that, like um the include and exclude, are both specified or um you know, there's probably a host of other rules that I know aren't validated like, for instance, build.m has a key in value, but you don't have to provide either and um we'll accept that um so there's a lot of like just you think about how a tamil could evolve over time and how it's really up to us like manually, validate those.

E

um So this kind of solves that problem, because you kind of have like a generic way to describe how it should look and you get errors if it doesn't match. So you get things like formats.

E

um I guess just kind of a I kind of jumped into the details there, but.

E

This is kind of like what the schema might look like you. Basically, you know we describe what it needs. um You know what's mandatory, what's not, we can require our name and value for like build in um that kind of thing.

E

uh It ends up yielding more of a you, get slightly better errors like you know previously, when you get an error like this, it says uh you know very goaling, specific error message and it's not super clear what it's about you. You get a slightly better error message. This is, you know this is kind of like a proximization of what it might look like.

E

Kind of one of the thoughts we had is it be useful for project tommel, since customers and users are going to be crafting this, but it kind of seems like we have a proliferation of tamil files appearing, and this might work as a good extension point for like defining how the other tombl files should be defined.

E

This one's probably closest to users. So maybe.

B

E

Good place to start, but something we could expand on.

A

Yeah, you might.

B

A

This in here, but it's also worth pointing out that we're doing this in javascript and not in go so it's that's part of the value of having it externalized from something like pac.

B

I really like this idea.

B

I was wondering if we wanted to put for all specified files which project tunnel is one of them like schemas and then also go bindings in the spec itself like this is what the oci spec does in a way that's convenient, and I think we could follow that pattern and then, if pack had other files that wanted schemas for those could live in pack or somewhere else, but I think having this for specified files in the spec would be a cool thing.

D

You mentioned go bindings. Could you elaborate on what you mean by that.

B

So in the oci spec, in addition to having json schemas, they have go structs that describe all the things that they have schemas for, so you can use them. You can consume it as a library. If you want.

E

B

E

Looked at those, but that kind of occurred to me that there are libraries where you can generate ghost structs or um I wasn't looking at typescript um types where you could generate them from the schema, because the schema is like just well-formed enough. You could do that. So I guess is it generated or is it just part of the.

B

I mean they commit the structs themselves to the project, but I think they.

C

B

Have uh tools in there to generate the strikes from the schema.

D

Yeah I'd be more hesitant to the go bindings option because it seems very opinionated to go where you know in some. In this case it seems like they're using javascript right and why pick go over javascript or any other language for that matter, that's where it seems a little iffy, and especially if we could have something like the schema itself being declared and having a different tool that generates those structs for any given language.

D

I don't see the value in the go structs.

B

I don't know I feel like the places we consume it in the project or all and go just because we provide those drugs doesn't mean we people can't use the schema in other languages right.

D

I thought you said in the spec and if it's in the spec I don't know exactly how we would consume it.

B

uh I think you can still import them like there's a go mod, I believe in the oci spec. Let me check whether that's true or not.

A

Yeah we could start putting like a package json in there and even create like a little mpm like from the spec uh yeah.

E

It's really interesting, while we're in there and just support every language.

B

Well, we could always just start with the schemas and leave the go part out if it's controversial.

D

A

That's a good thing about it.

D

A

Other good thing about putting in the spec that I didn't think about is you just naturally get the versioning of it. You know if we make changes to the spec, you don't have to like go into some other repo and make sure you have all the tags right and stuff.

E

um And I know there are tools that exist that can generate uh documentation. So if we want to like add, like you know,.

D

E

D

E

Bit hard to read, you can actually like generate like a you know like a markdown document out of this that's human readable. If we feel like we need that.

B

That's also why I like the ghost trucks when I go to check oci things. I read the ghost trucks because I find them readable, but maybe not everyone agrees that that is the best ux.

A

Emily doesn't read books, she just reads: go code.

E

All right cool, any other questions or thoughts.

E

D

This is this sorry. I guess I just wanted to kind of summarize everything that we've said so far, so this rfc is to propose a schema for project descriptor, but it sounds like we want to overall, like I guess, increase its scope. To just add it to any schema or any tamil file in the spec itself.

D

E

Yeah, I think that's what I heard.

A

Yeah, I mean definitely the the value for us is from project tamil, um but I don't know I could definitely see wanting the other stuff specked out, especially as like uh more people start calling lifecycle directly or it'd be nice to have one for builder tomel too.

A

B

A

B

Metadata tomml is another one.

A

B

Different platforms are going to want to write to give to the life cycle.

D

And as we're kind of like already generating some aspects of our documentation on the buildpacks.io site, I could very easily see the schemas being a way to produce. You know that auto-generated documentation for the different configuration or descriptor files.

C

E

Inside the json schema, so you could actually like have.

C

Readable bits in there that hopefully.

E

Get translated to them.

A

uh So because project tamil is already an extension, I think we can start with project tamil really easily and just add it to that same tag on the the spec part of it. um So it'd be a good place, like that's a good way to just dip a toe under the water, and if it looks if it works, we can expand it to the other tamil files.

B

It's worth noting that this fights against uh the idea that the key name in project tommle can be derived from the file name. I don't think he can make a schema that describes that, but I would feel a little iffy about that suggestion anyway. So I'm okay with that, but I thought I'd call it out because it's in a different rfc.

E

The key name, I guess I'm not aware of that one, the key name being a wild card.

A

Yeah I like how, for us its function and for project domino, it's project.

A

Is there a way to do that in json? Schema just have like a I mean I like, like describe the schema of the sub section. Like I guess you, I guess you could just like scope. The schema to what's inside of that table.

B

Yes, like you could do like a wild card, but to make it valid based on the proposal in rfc terrance opened, it would actually have to match the file name, which starts getting weird, that's sort of why I don't want the file name to affect the schema at all. I think it it's awkward, you're, muted,.

A

I think we can have that discussion as part of the uh that rfc and not this rfc, because I think.

A

Cool thanks for sharing that josh.

B

Did josh leave.

A

Hi josh, oh, he did.

D

A

B

Mic drop. Everyone likes it, I'm out.

A

uh Yeah, actually, the timing of that was funny um yeah. I uh I guess, since there's nothing else on the agenda, I made some updates to the app mixins based on our discussion yesterday um and I'm feeling like it's pretty pretty darn close. uh I want some of like the key names and the schema I'm not super happy with, but the actual structure of it is what I want so take a look when you get a chance.

D

So I am curious um in relation to this it yesterday you mentioned that you were already kind of like prototyping, some of this to make it work and see how well it all ties together.

D

I'm curious how that ties into some of the other efforts that I know I'll be probably focusing on next week, which is the ca cert stuff, and I know we kind of um created a little bit of a a plan, a game plan to tackle that was to enable the volumes the rewrite volumes, which is easily attainable. That rfc just went through and then start actually implementing some of this ahead of time.

D

So uh yeah, I'm curious, like what that entails. uh Is that some.

A

Are you you should keep doing that.

D

Are you gonna continue to work on this stuff, or should I work on it.

A

I'm not going to work on I'm not going to work on the case or anything, but.

D

You are working on the stack, build pack part which I will be kind of re-implementing to try to get the cacer stuff to work.

A

uh I don't know how you're gonna do it, but um I think, like our like, if we end up with a ca, cert stack build pack. um It would probably leverage my like my hunches that would leverage the same volume mechanism to uh to bring in the certs right. It's just sort of a different mechanism.

B

I guess my question javier, is: if you're going with the, if you're, mounting into etsy ssl certs that seem sort of orthogonal to the root, build pack thing right.

D

uh It isn't so. um I don't know if you saw a reply to your comment where the read write: volume works as a as a workaround right as a hack. I think you declared it a hack um so that I don't think that's the problem right. It's kind of the next step the evolution of that is.

D

If we wanted to actually provide a stack, build pack that when you placed a search file somewhere right uh that, then it detects that and executes the update, ca search automatically so that you don't have to provide the entire contents. You could just provide additional certs that get added to the store.

D

That part, I feel like that's where the stack build pack comes in and from my mind, there are some changes that you probably already did joe to life cycle to execute the stack, build packs, and I don't want to have to recreate it. If that's.

A

Something you've already done so, first of all, it's very early like we, you know we were just proving out the the caniko stuff right, uh there's still um I'll, probably jesse's, probably gonna pick it up where I left off- and uh I know he's out next week. So the timeline like I do not want to attach the timeline uh for the for what you're doing with ca certs to stack packs right now. um I think that would be um just a disservice to the community right.

A

So as long as you're, okay with us reworking that later on potentially like- and maybe we don't- maybe we just say yeah whatever the thing pack's doing it's fine other platforms can use stack packs for this for this capability. um So like. Let's, like I guess, what I'm saying is: let's just work on these two things independently, as though they aren't connected. Even though we have a feeling that we will use stack, packs down the road as a general mechanism for certs.

A

Does that sound? Okay.

D

It does with the like the idea of saying, like hey, we might stump on each other's toes and we're both okay with that and we'll figure out how to essentially merge that piece of work. The part where maybe I'm a little confused and might be what emily was trying to get at was like my step from here, would be to use uh stack, build packs. There is no pack, only implementation that I can see outside of the read write volume mount.

D

A

So you're actually pretty much yeah. Why.

B

Don't we just leave the the volume hack until we get the stack, build packs right.

D

Well, I mean we have it already right like that, would be implementing the re-rate volumes. There is no, unless I'm misunderstanding, what you're saying with that phrase.

B

I'm saying to do the real version of it. We need stack, build packs, so it's not. We can't do it until there are stack, build packs.

D

Right and I guess my effort was- or my idea was to put effort behind that to solve for the ca certs like actual implementation effort.

B

Like into the the stacked packs, the life cycle running stack, build packs all together.

C

A

I think, like, let's see that's too bad, that jesse's out next week, because I didn't want to just go: do it without him, but we could like. The branch is up there like you, could pull it down and you know pretty soon start to play with it and try to build something on it. Like a actual cs3 stack build pack, but uh I worry that that's gonna just delay some other hack that we could get out for people to use.

D

I guess that's the part like I'm not sure I I see that other hack.

A

Yeah yeah I'm just making an assumption that that.

D

There could be another.

D

A

uh Yeah, I don't know, um is there like, how much can you do in pack before you would need like them a proper mechanism from life cycle?

A

Like could we converge at uh in a week or something like that.

D

Yeah, I was gonna say like I don't know that it has to be next week right that I I start working on it. I could definitely wait longer.

D

The the only rush I have is just the importance of the ca certs, but nothing specific that says: there's a deadline right.

A

D

So I just posted it in the chat, and I led it to the document as well kind of like the exploration I did and the quote-unquote solutions that we currently have or will have so one of them is extending the builders.

D

That's what we currently have the use of volume mounts is what we will have when we have rewrite volume mounts that could be placed anywhere um and, like I said the only like, I can't find a solution between the volume mounts and the stack build packs right without it being tied to a very specific operating system in, like a really really hacky way.

B

I don't know if you saw my comment about this in slack, but what this example on the gist is doing is mounting from like a search directory in your working directory, but I think if you actually mounted etsy starts to etsy certs, it would mount the ones from the docker vm, which I checked and actually contained the ones from your system system search store.

B

So in a way people wouldn't have to do really any work at all. They would have to create a set of certs and put them together. You could just mount in your system, ones from docker.

D

I definitely did not see that that was three days ago, so you're saying that there are certs elsewhere.

B

It's in the the docker vm has a search store and I check the documentation. It claims that it populates it using your system search store. So if you install a new cert on your workstation and then you restart docker your new search in the in the docker vm and then, if you mount, if you said dash volume, etsy, search or etsy ssl, search colon at cssl certs, then because that's not in one of docker's allowed share paths, it will actually mount the directory from the vm not from your host system.

B

So you can kind of just make it magically work with all the things that are on your host.

D

Okay, I could definitely explore that. I was not aware of that as a possibility and.

B

I guess that could be.

D

The interim yeah sorry, I.

B

Think that's actually pretty good like it gets you through all your build problems, and you can even do it at run time because it's not specific to us at all we're just documenting a feature of docker, but I think as long as people are using pack and docker locally, this should solve their problems.

C

I mean, I think, there's a use case for you want to bake. Ca starts into the final image so that you can portably run it across different platforms or you're. Not you know, you can't.

B

Don't have that problem.

C

um But, but to solve the build time problem then like for pack specifically strongly agree.

B

I think it solves a big category of problems at least, and then that makes me feel okay waiting for the real stack pack, rfc spec pr lifecycle parade to progress before we get the complete solution.

D

Yeah, and so this solves for build time right, it.

B

For runtime and docker you can do the same thing at runtime, but.

D

It doesn't solve.

B

For portable runtime solutions.

D

Okay, that's that's basically what I was really looking for right. The only way that we can solve portable runtime would be. uh I mean if we even wanted that I think really is a stack, build packs.

B

Yeah, but a lot of people will then run these images if they're thinking about running them, other places would need the search, they're running them in case and there's a pattern of using init containers to install search there anyways so right, I'm not saying that nobody wants to bake them in some people definitely do, but I think there's a lot of people who for whom this would be sufficient.

B

I think the big gap is ci. You would need to uh install your shirts in ci as well.

D

Yeah, the ci's that already use pack and docker that'll work.

E

As we say, if you're already using pack, then you can just uh create a volume and do that.

B

Yeah or you need a solution nci to install your search anyway, it's like, if you're running this build without pack, you would still need still need your shirts in ci.

D

Well, I'll definitely take a look at this um and yeah. Hopefully another solution comes out of it in the interim for before stack, build packs and then we're not stepping on each other's.

B

B

So, are we done early.

C

uh I I added something to the end of the list when I joined just, I updated the rfc around profile d to use a totally separate directory uh and um kind of better defined the formats for the output and added a prior art section on joe's request. um Just main thing I wanted to ask, is uh you know joey you brought up last time, you know. Is this a pattern somewhere? I don't know when I thought back.

C

I didn't remember if we actually addressed that or if we changed subjects somehow um do you uh do you still feel like uh it needs more investigation into the sorry.

A

No, I think I approved it with the exec d thing: cool.

C

Awesome, the format is just the list and there are, there are some things like uh windows use the strategy for adding up to the environment. So I just.

A

C

Make sure I'm comfortable with it yeah.

A

Yeah, no, no yeah, it looked good after you made those changes.

C

C

That's all I got.

B

It seems like that toronto of new rfcs you added stephen most of them- are ready to move to fcp.

C

uh Given terence is out this week, I'm I'm happy to wait a little. You know until he has a chance to look at them. At least if you know that'd be a I'm kind of looking at joe.

A

Yeah I'll uh yeah: let's do let's wait till monday. If that's fine I'll just grab them on monday and I can go over them all with them. I think it'll be pretty straightforward. So as long as you can wait, two business days.

C

Happy to want to make sure he has a chance to provide feedback. um I think they're small, hopefully and not not super controversial things, but I definitely don't want to move too quickly.

C

A