VMware Carvel Community Meetings, 19 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Carvel Community Meeting - August 19, 2021

Description

Carvel Community Meeting - August 19, 2021
We meet every Thursday at 10:30am PT. We'd love for you to join us live!

Full details on this week's meeting here:
https://hackmd.io/F7g3RT2hR3OcIh-Iznk2hw#August-19-2021-Agenda

A

Hi everyone welcome to this week's edition of the carville community meeting. Today's date is august, 19th 2021, if you're tuning in in from home, watching this as a recording, we encourage you to join us. Live we meet every thursday at 10 30 a.m. Pacific time it's an opportunity for you to come meet the team you can lurk in and just listen in on what we're working on or, if there's something that you need help with.

A

Regarding any of the tools, the maintainers are eager to listen in and provide feedback and help you through that or if you have any feedback on how you want the tools to be improved.

A

We also would love to hear from you regarding that as well. If you aren't able to attend any of the meetings live, you can share with us through a couple of other options. We have uh slack and twitter kubernetes slack workspace and the carville channel, if you're, not part of the kubernetes select workspace you'll just need to request an invitation to join. It's really easy.

A

You'll just send the request and they're pretty quick to approve that request.

A

We ask that when you attend these meetings too, please read and abide by our code of conduct and add your name and the organization that you represent here in the attendees section. That helps us understand who, outside of vmware, is attending these meetings and to keep those lines of communications open.

A

Just a couple of announcements we have here, we released ytt 0.36.

A

And does anyone want to go over this release.

B

I know if garrett was here: he'd have some details. He was a big part of that. um I think in general. The theme here was improving uh error messages that are coming out of our newly ga schema features uh and a few kind of internal uh cleanups um uh yeah. So these these are some follow-ons from from that scheme of work.

A

Great thanks, john uh next up uh just a reminder to register for the spring one workshop that we have coming up on thursday september, 2nd. If you are wanting to learn more about carville or you know, you're still, maybe just using a couple of the tools that want to learn more about some of the other ones. This is a good opportunity for you to get more hands-on with learning how to use all the tools in the carvel.

A

Okay on two status updates over the project roadmap itself, we have private registry authentication for cap controller. How is it looking with that release for that item?.

C

um Things are moving along pretty well here, uh I think, for the most part, it's been, you know, daniel um joe eli and demetri that are are focused on on some of the stuff, but we we got over kind of the.

C

What I would consider like one of the the big initial hurdles, which is just kind of like a a steel thread of the package, install experience um so just just wrap that part up and just continuing to keep keep focused on this. I don't know joe dimitri anything else that you want to add to this.

C

A

Thanks aaron and then the other two things are coming up for september.

C

Yeah, so the the second one, um I think we actually have a discussion item to go over the the proposal for the exporting ytt schema as an open api schema, so we'll probably chat about that a little bit more. um The third row around just api package management api enhancements. This was for like a fairly generic placeholder, and we should probably update this, but nothing specific to call out. There is just like general ux improvements, um and this page was less updated last month.

C

So we should probably update that update this soon, as we've started to have some more planning discussions recently.

A

Cool thanks erin, and um will you be updating that, should I put down an action item.

C

Yeah, that sounds good.

A

uh Next up the backlog is there anything um within this that the team's working on this week that you want to call out that's outside of the roadmap.

C

uh I would maybe just encourage individuals if there's anything that they want to share about what they've been working on this week.

D

um I can give a small update uh me and garrett have been working on uh the story to capture image, meta's information in an image package lock file in k build.

D

um This was a story I picked up a few weeks ago went on vacation, so it is, it is getting attention once again um and then yeah otherwise. For me, just working through the open api proposal.

E

One thing that uh I worked on recently and actually failed to uh let nancy know, there's a new vendor release um that included a few collected items. um One of those items actually was a.

E

Support for automatic tag selection um when fetching things like an image or image package bundle from a registry. There was also a few other enhancements done by the community members that are called out in the release notes in vendor. um So that's something that's something to take a look at if using that too.

A

Thanks, dimitri uh yeah, I I actually saw um I was tagged in the github one and I'll make sure to tweet that out that that release happened.

A

Okay, anything else that folks want to share.

A

Okay uh onward to discussion topics and again, if you have anything that you wish to discuss with the team or share with the community, please be sure to add that in this section here and we'll get to it, first item is something that I added in here wanted to bring this uh to the community. We are calling on y'all to share with us if you're using carvel, we would love to hear from you first off. Thank you for using karvel. um We want to hear from you. How are you using it?

A

How can we improve it if you would like to be featured on our doctor's file? That is an opportunity for us to help promote your organization and continue to work with you.

A

All you need to do is go to this pinned issue and share a comment with uh the following information: your organization or company, the link to your website, the country, uh your contact information.

A

What is your scenario for using karvel? Are you running the application and testing or production, and then attaching a png version of your logo?

A

We already have a community member that has filled this out already thanks to scott from terra sky. I really appreciate you adding that information in there, so it's a good example here of how to fill that out.

A

Okay on to the next discussion topic: the open api proposal, I'll stop sharing my screen so that whoever put this in there, we can use that.

D

Yeah I can show my screen.

D

uh Cool, so um for this discussion topic, I was sort of hoping like to give a brief overview of the proposal and just a few parts of the proposal, specifically parts that haven't been yet decided.

D

I would love to get others feedback on what is currently being proposed and if you know others see any issues or have alternatives that they would like to mention, and that would be a great time for that, um and also like feel free to interrupt me and ask questions about the proposal as I kind of go over it or bring something up if you're like. I don't know what that word means, or you know something like that. That feel free to interrupt.

D

So this piece of work is to generate an open api schema from a ytt data value schema.

D

It's specifically in the open api format.

D

This format can be used for validating inputs, using like other tools that do a open api validation. It also serves as documentation to what inputs your ytt configuration allows.

D

So you can supply information like a description for.

E

D

Key value, the type that's required.

D

um You know some validations like a maximum number or minimum number, that sort of thing um ytc right now has its own data values, schema format, and this proposal is to use that ytt data value schema format to generate an open api, schema format.

D

So the current proposal introduces um two flags, so the current ytt flags look like this: there's a files, inspect data values, inspect uh data values and spec text and that show json to change the output format.

D

Following that similar flag format, we are proposing a data value, schema, inspect flag which will print the ytt data value schema in the open api format.

D

That's specifically this flag, when you use it with dash o open api, there's also a flag for just viewing the schema in ytt format and then there's also a flag for viewing the schema in open api full format. So this would be not just the schema portion of the open api document, but also including the headers and metadata that that comes with that, such as version scroll down to so what that looks like that's these additional fields, open api version and info, where you can give a little bit of a description about the document.

D

Otherwise, just the schema section would be everything under components.schemas which might look something like this.

D

Any questions so far on that.

F

um Just a clarifying question in the current flag set: uh is there a missing space that last one ytt data values inspect dash? Oh, is it is it ojson? All is one thing.

D

Yeah, I believe that you can do dasho space, json or.

B

You can do.

D

It all together, as one word.

E

E

Yeah, because the show takes uh you know, takes an argument. uh Cobra is smart enough to understand that whatever comes next is for their show now for booleans, not gonna work.

F

Okay thanks, I just want to make sure we weren't introducing uh inconsistency thanks.

C

One other thing I would just add on that flag in particular: there's there's other types, yaml, json or pause, pos and yaml is the default value. So that's kind of like what the second one is.

D

Yeah good call out.

D

Okay um cool so now that you have sort of an overview um I'd like to talk about how this would be used specifically in cap controller. So cap controller is one of the users of this open api, spec or open api schema and cap controller has a field in the package cr. So this is cap controller stocks, there's.

D

A section in this package for value schema open api v3. This is where you would put a ytt open api schema that documents. You know what values your ytt configuration allows because this package uses and templates ytt configuration. This is a way to document like what is allowed.

D

So, cap controller, being you know, one of the users of this has sort of a specific use case. So um in the proposal there's a recommended workflow of how you would insert that the open api schema into a cap controller package cr and that workflow is fairly straightforward. um You know you'd create the schema from a data value. Schema file might look like this.

D

You'd save it to a file, maybe name that file open api.yaml.

D

And then the way that you can insert it into the package cr is using ytt. So you can just add the open api file that you just created as a data values file using the data value file command.

D

You have to name the data value um using this method, so call it open api schema and that's what this syntax is and then in your package, cr, all you need to do is import.

D

You know a few libraries here, data and yaml, and then you can use your data value, open api schema right here and it will insert this file into your package. Cr inline right here.

D

Another thing I want to call out is because the data is read in as a string using this command. You do have to do this yaml encode, to make sure that it's in yaml format, rather than in multi-line string format,.

D

um And so the reason I bring this up is because you know this is what we believe is the simplest workflow for using this open api schema with cap controller and I'm curious to get feedback from. Maybe those who work on cap controller on whether there's anything about this that you know doesn't make sense or would be you know a rough edge or you know anything like that. Anything. You notice I'd love to hear from those.

E

E

I think this is a fairly um fairly, I think, good attempt and how we can insert this. uh You know this chunk of yaml into into another chunk. One unknown I have in my mind is this is something that we haven't really firmed up, as a recommendation from the packaging team is how does one actually kind of uh what's the workflow that we recommend for producing and storing the package crs right?

E

So currently we just say produce a package cr and then you know include it in your repo bundle um where it's going to get used by. You know some other tooling right. uh We we don't really recommend right now to at least try to template the package crs right, because you know once you introduce templating. That means that you're kind of maintaining a live set of you know uh of configuration versus we see. Package cr is more like a record in time right like once, you create it. It's there.

E

You you shouldn't probably be you know, messing around with it or regenerating it or something like that right, and so you know, I guess: how does that relate to what's kind of uh the example that you're the example workflow that you're providing? Is that? Probably then you know this package yaml really acts as that starter right, and then the output of that ytt command would be like. Okay, here's the here's. The package here that got generated, save it right, save it into the git repo directly right.

E

So don't don't keep on regenerating anything, so that's definitely a viable way to to to kind of uh glue those things together right. um You know. Obviously you know everybody has access to bash, so they can try to glue it directly in bash without uh without ytt, but that's probably uh not ideal, since uh you then have to figure out to indent certain things and whatnot um so yeah. So this this is, this is pretty good will.

E

Probably this is more what to do for the packaging teams, we'll probably need to circle back and actually provide some general recommendation. We've been working with a few um vmware teams that are consuming this packaging format, so this is the this recommendation would probably be very helpful to them.

D

Yeah, that's a that's a great call out. Like um you know, package ceos right now aren't normally generated using ytt templates, so that would be a bit of a different workflow than his current um yeah.

D

Yeah, that's interesting, um because we definitely want to recommend that you know this doesn't get regenerated. If a ytt data value schema file gets updated right because I think a package cr is like a tied to a version right, so it doesn't necessarily get updated.

E

D

E

Yeah, this kind of gets back into a question of uh what is that system that uh people use to keep track of this package crs right? They might have a ci pipeline that says, okay, this, this version of a package is good, we'll publish that up to the uh to the to the registry, and then we will, you know, have this package cr dropped in into some. You know folder in a git repository or something like that, and so I think actually now that I think about it.

E

You know the other pieces in the package here are like a version, for example, need to be inserted in right or, for example, uh the reference the digest reference right should be placed into the package cr at the time of the creation of the package cr, so there's other pieces of information that need to go in so this would be.

E

I think this would be in line with the with the open api schema as well.

D

E

Maybe then this boils down to should we be providing like a little common template that you know folks kind of a base of their? I don't know ci pipeline on or something like that.

D

Yeah yeah. That makes a lot of sense that that definitely seems like something that uh maybe you know, the packaging team wants to kind of standardize like it seems like this templating or this proposal for how to insert the open api schema is just one aspect of that, and there's probably something larger, that that could be done for like the entire package cr document.

D

This aligns with that. So.

E

Cool yeah yeah.

E

I have a by the way- I don't know if uh this is a uh this is an accident here, but it seems like the indentation disappeared on that package cr for some of the things I don't know, if it's just the browser uh bug.

B

E

B

E

I've been beaten by uh github and firefox now working this week very well with indentation. So I don't know.

G

I have a question on that key though opening open api v3. Does that mean there's a v2, a v1, potentially a v4 like in the future, and I'm wondering how that ties to how we generate the specific version for open api, and I apologize if it's already been discussed in the in the issue or pr.

D

Yeah, I think only open api v3 exists in a package cr.

D

um So that's to say, I think open api v3 is the only supported format in your package here.

G

Yeah and for other, I really like this question.

E

Okay, go for it dimitri.

E

I really like this question because it it poses a you know, should we be actually using v3 as a suffix in the generation side of things right because, let's say when open api v4 comes out right, what is dash o open api really mean right, and I I'm guessing that's where you were going at uh yeah the generation part of it yeah.

E

Maybe, as a side note, the reason why we picked open api v3 is this something that kubernetes uses in the crds as a key as well, uh so we're trying to be aligned with that, um but yeah. I think uh I think when we would finalize the the show, let's say naming right: we should actually be more strict about what what is the thing that we're generating here.

C

I'm mostly guessing that this is something that you get from the metadata. If you were to do like the open api full, not just the schemas like it would tell you which open api version you have.

C

So I don't know that that could just be more things to consider if we made things a bit more open, but you also required, I guess, a version to be provided as well.

E

From a packaging perspective, one reason why we really want everybody to to be using the same things right is we don't want the clients to be struggling with trying to figure out? What are the you know tons of possible variations right? This is a similar kind of uh uh like you know. Originally, you know there were questions around like well. Should we be, you know exposing the ytt schema here, but maybe for packages that use you know helm template underneath.

E

Should we be somehow exposing whatever you know, some other format there right and so the thing here is that the packaging api should formalize or should normalize.

E

In kind of asked so that the clients are not necessarily, you know trying to support all possible schemas, but over time I guess you know, I think that's that's the intention here behind this key naming in in kubernetes as well over time there will probably be other versions that are supported and they actually might have to be filled in you.

B

E

Both versions right to to be backwards, compatible potential.

E

H

D

And I know that, like the open api version between v2 and v3 is really different, like a lot has changed, so uh it seems like an important distinction to to say what format it is.

D

um Normally, when you have, when you have an entire open api document, it includes the open api version. Specifically, however, the cap controller and the kubernetes crd only includes the component that schema section of that document, and so it emits the open api version, which is maybe why they chose to inline it because they don't include the entire document. There.

D

But I think that's yeah.

E

They end up, they end up generating this, as like. A full document is being generated at a particular api endpoint, so that uh you know they're in control of uh you know the actual surrounding other information.

E

So that's why it's kind of it's it's the chunk and you can kind of argue from our perspective, even though we're not actively doing that. We would potentially want to do something like that as well.

D

I think that also brings up an interesting point of like this may work well for cap controller who's consuming this uh open api schema.

D

um You know: how will other people be using this open api schema that gets generated like you know, do they need to know the version that was used in order to create the open abs schema? um Because it's not like in the current proposal. It's not included the version information so, for example, um the open api scheme that will get generated looks like this.

D

It's just the schemas section, so it doesn't include the version which would look like this if it did include the version um so yeah, if somebody was just outputting this to a file to use as like documentation for their ytt configuration it wouldn't it would not include the open api version or anything. Other information like that that you know people may want to see um by default. um There is a command proposed to include all of the headers that includes a version using this.

C

I think maybe I was making some assumptions from this conversation, but I uh I I was taking away that we would update the flag from open api to be open api v3 and then have an open api v3 dash full as well, but maybe maybe I misinterpreted some things earlier.

G

hmm That's what I was going with it, that's what was my thought. You know when v4 comes out and they want to use a v4 generator. V4 schema will be open. Api v4-4.

D

Yeah, I don't really consider that, but I think that's something that we should make note of.

D

Something like this.

D

Yeah, I think I think that makes a lot of sense. That's actually something I wanted to talk about next, unless there are any other questions on the cap controller workflow.

D

Cool, so um we haven't really come to an agreement on what the flag name will be. I think there are a lot of good options out there. um The one that's currently being proposed is like, for example, in this uh scratch pad that I have here this, the the ones that are being proposed here.

D

Data value schema inspect, which takes a dash o open api flag.

D

um This command um kind of plays off of the command that we have right now. Data values inspect the way that command works is all of the data values and schema data value schema files that you input into ytt.

D

Get output after they have all been merged together.

D

And the command that's being proposed is similar to that, but it only well.

D

It would be, it would also include the annotations that are present in a data value schema document like, for example, the annotations exist right now are um like pound at schema uh type any and uh schema slash, nullable those sorts of things. That document exactly like what the schema document allows.

G

What about in your ytt project, if it contains a library and that library contains a schema, does that get output as well in this open api document, or is that just emitted? What was it multiple docs get sped out.

D

Yeah at some point I considered a little bit. I I think I think it makes sense to omit them, at least for.

D

The first iteration of this.

D

But you know, because you can target a schema at a library you can use a library, ref annotation in your schema, to apply a schema to a library. I think it's something that should be supported with this command. Eventually.

E

Yeah there's a I forget what we do for data values, but I'm pretty certain. We don't go on and kind of, recursively show that information, uh mostly because you know when you're, ultimately working with the root library, right and so you're trying to figure out. How do you configure it now? There's obviously some use cases where you may want to configure.

E

uh You know a library instance: that's been instantiated by the root library, but it is today we don't, we don't expose it. If we would expose that information, I would imagine, then we need to have a more general way to select which library you're looking at right. It wouldn't be like specific to data value schema. It would be specific to everything right, like maybe the entire output, maybe the the the data value well, not maybe, but for data values for schemas for the output for overlays all that stuff.

E

It's almost like you can think of it as like evaluating only a portion of what's given to you.

D

Yeah, I think it makes sense to follow the same pattern that we do for data values inspect right now and so yeah, since that doesn't include library files. I think it makes sense to do the same for data value, schema, inspect.

B

It also doesn't feel like it's too onerous on users. If you want to see the schema for a library, you can target your ytt invocation to that library and include this flag and you'll get the schema for that library, because at that point, that invocation will see that library and its data value schema will be the subject the root um and, if you're, if you are using uh library ref to provide data values to a um to a dependency of your root library already, then it's already a separate file.

B

You already have this kind of experience of your kind of reaching on a side channel. You have that feel of you, you're, reaching on a side channel to provide that configuration.

B

This doesn't seem to introduce any awkwardness either into that flow. For now,.

F

That's the point.

D

um I've listed some other flags that were considered and proposed by others, um and you know if anybody feels strongly about any of these ones that were proposed like feel free to to bring it up.

D

Now um there were some that uh use the word generate and create which uh I I think it's a great idea, but does not follow the current pattern that we have for other flags and following the same pattern, is one of the main motivations for why I chose the data value schema in spec command because it mirrors the ones that already exist, um but I am happy to consider ones that use the word create or generate more, if others feel strongly.

C

I I was thinking about this a bit because inspect feels, like you know, it's retrieving information that currently exists, whereas we do kind of create an a bit of new information from existing information. So, like I thought about it a little bit, but ultimately I'm fine with it, but just wanted to share that thoughts. I don't know if it resonates with with other folks or or even like the current usage of the existing commands and flags.

E

One thing that, on the side, we're discussing with john is kind of at a high level. What are we like is there? Is there some um how you call it a more holistic way to look at this stuff right? If you're going to kind of imagine ytt as a pipeline right where, throughout that pipeline, there's the schema evaluation, then there is the data value evaluation. Then there is finally the templating and there's finally the overlaying and then well that's the end right.

E

We are inspecting a schema at a particular point right at the beginning of the pipeline, because it's the first thing, the data value schema specifically not just journal schema, and I think I know we were toying around with kind of different ways to really think about it. I think, at least in my mind so far, what I have is we're just presenting ytt schema in a different format.

E

Now, what does that mean to use dash open api? Let's say in a different context, that kind of gets into a very hypothetical world like what is the input to that right? Like typically, it's ytt schema. But what is the input you know to the other thing? If you're, I don't know, maybe maybe you're saying you see- data values in spec dash open api. What does that mean, um but anyway, so that that's kind of why I'm leaning towards inspect is because we are just inspecting part of a pipeline?

E

Maybe there's even a greater argument to be to have. Is that, should we have a general inspect flag that takes in an argument like data values, data value schema?

E

I don't know uh post templating, that kind of thing right, but but I think, if we're going to generalize some of that stuff, then we should probably have a more uh uh uh more complete view of what we actually would want to to show like, for example, should we be allowing people to see the output after each overlay, for example right that kind of thing, which would be really handy if you're actually seeing it like after this overlay? This change happened, but then, like you, know, click on the next button.

E

Here's the next change happening on the next overlay, so it would be really useful for debugging, but also kind of uh consolidates this bunch of functionality into a more general feature.

D

I like the idea of having a generic inspect flag, but I also like the fact that they're specific to what you want to inspect since uh it allows you to just invoke ytt with an entire directory, and you can just get only a specific part of the files you gave to ytt in that inspect.

C

I I do have one other quick question that might have been addressed in the proposal. I haven't looked at it in a couple days, um but the flag, I guess the open api and open api full.

C

um I'm curious what the thinking of here for open api to be just the schema versus kind of like the full, and then we would have you know something like open. Api schema for just the schema, and my thinking of that is like open api as like the default option, would return what you would expect from an open api. uh I don't know what the output is format or document, whereas the schema itself feels like that's something, that's more specialized but curious to get thoughts. There.

D

Yeah, this is a great question, so I like the idea of optimizing for who the users that we know that we have right now, which is cap, controller and cab controller, really only needs the schema section of the document.

D

So that's one of the reasons why I decided to choose it as kind of the default open api format. It's because we know that people at least one user wants that right now and I don't have a great idea of if you know other users will even really need the entire open api document and so making that the default format.

D

You know, I don't want to add additional information to the output that may not even be needed. um So that's kind of that's kind of my reasoning behind it, but yeah curious what I think others think.

F

I could offer um I I don't think of cat controller as a person. So in specifically, I think cap controller gets to kind of hard code, its own arguments, that it needs somewhere and then never cares again if they were wordy or long right versus. uh I think a human who stumbles along a few months from now um is maybe the more interesting case to optimize like their ergonomics or their experience.

F

uh Even though I appreciate that you know we all get to be good friends on the carvel team and look out for each other's uh use case. uh It actually seems like one of the easier ones like you could have write the longest ugliest flag and we'd only see it like every six months when we edited that file.

E

I I think what kerry meant is the users of packaging apis, because cap control itself doesn't actually execute this command.

F

Okay, sorry, I misunderstood.

D

I still I still hear your point of like you know there can be extra information and if cap controller doesn't need to worry about that, then it's sort of inconsequential.

D

If I understand you correctly, providing extra information is not necessarily detrimental to the cat controller use case.

F

Yeah, but I think I I thought cap controller was directly invoking it. So if it's, if it's humans in all cases, uh that's a little bit uh that I think that negates what I was trying to say.

E

Is there is there? um Is there open api terminology, for I mean there's, obviously a term you know the key that they use schema as a you know, as a key for that section right I don't know if there's a open api terminology for the entire document, maybe it's open api document. I don't know that.

B

Is what they call.

E

So that might be that might be an alternative right, having a open api, doc and open api schema.

E

Arguably, maybe cap controller should have actually called it open api v3 schema um instead of uh you know, well not doing that.

D

Yeah, yes, your point.

B

Yeah, I feel like this question was a was a bit about deciding which bias to choose. I like the way that you were describing how you were thinking about the decision, carry. If we look at it from a carvel user, then they already know they're generating schema and it can feel a little bit redundant in one's mind. If I'm saying data value, schema, inspect open, api-schema.

B

But if I instead place my bias from the concept of open api from the world of open api and I'm naming the object, that's getting generated, that's exactly what dimitri was pointing out. That, like the output, would be um in fact, probably open api without a qualifier.

B

I might assume coming from the open api perspective, would be a open api document and that the dash, I think, aaron you were saying this, and that the the dash schema is the special selection uh within that document for the schema component.

D

Yeah yeah, I think I think that also makes sense. I think we should dig more into that of you know what users expect from this command or from this flag. um You know: is it just cap control or is it you know to users? Have other needs um we're out of time today to talk more about this, um but this was extremely helpful and thank you all for sharing your thoughts on this and asking questions um hoping to continue this conversation in the pr.

D

um So please, you know, feel free to add comments here, or you know kind of a follow-up call for this.

A

Thanks carrie for bringing that up for discussion and thanks everyone for joining in if uh you're watching this from home- and you would like to provide your own thoughts on this, please feel free to drop those comments in that doc that carry shared uh or find us on slack or twitter.

A

um We would love to hear from you and also just a reminder, like I mentioned earlier in this meeting, we would love to hear about how you're using carvel, so please uh go to that pinned issue and share your details about how you're you're, using any of the cardboard tool, suites tool tools and the tool suite and with that um have a wonderful rest of your day. Thank you.