VMware Carvel Community Meetings, 25 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Carvel Office Hours - March 25, 2021

Description

Carvel Office Hours - March 25, 2021

Details here: https://hackmd.io/5Bh2IXwTShSrA0YdBY4AGg

A

All right, hi welcome to uh the second edition of carnival office hours. This is being recorded and um will be uploaded to our carnival youtube playlist, a reminder to please read and abide by our code of conduct while attending these meetings and the purpose of of this.

A

uh This office hours meeting is just to have an open space for discussion topics and anything that you or you know anyone from the community needs help with regarding karmal or if you have any questions regarding the tool suite or would like help getting started with any of the tools in the carville tool suite the maintainers are available in the office hours to help with anything you may need. Regarding any of that. So, first up on the agenda today, we have.

A

Bring back support for starlark multi-line.

B

Strings and helen, I think you said you yeah um so as aaron and I were kind of going through different um get-up issues in the backlog.

B

uh We noticed this was open six days ago and just wanted to bring to the table to the maintainers and see if there's something that y'all wanted to discuss about and decide on whether to bring back that multi-line.

C

Support, I can provide some context behind this for those that may not be familiar with the issue, um so I had a question on how to use starlark's multi-line strings. Like this example here, these three quotes signify a starting of a multi-line string and the end of it, and so all of the white space in between, in that example, starting with asd 1, should be preserved. So there'll be a new line and the next line and a new line and then like a number of spaces.

C

Currently, the white space gets removed when using a star like multi-line string and results in um everything being indented the same amount.

C

So that's what you see in that generates example.

C

um So this is because, several months ago we encountered an issue where, um when having a, uh if I'm very quickly, it's a like new document, start in between two segments of starlark code, the starlark parser detected an indentation error, um so we use a modified version of starlark in ytt, which strips white space. Hence why? When you start a function, um you have to end the function with the end keyword.

C

So we wanted to basically remove starlark's uh indentation rules, since we, you know, use starlark a little bit differently and uh as part of like removing that indentation error that happens from starlark, we removed the ability to have starlight multi-line strings because we removed the white space.

C

So that's where this came from this was something that we knew about when tackling that that issue it was considered and considered in the solution to be a reasonable compromise.

C

Don't have all of the details on on why exactly we we chose to do that, but um yeah. So the story is then to add that functionality back.

C

Does that make sense any questions.

B

Yeah thanks for that carrie, you know I just wanted to get a feel from the maintainers versus maintainers, whether this is something we want to support sooner than right, rather sooner or later so um yeah, that's kind of like the discussion. I was hoping that um you know or a conclusion that we can get out of through this discussion time.

B

If we think this is not important, then cool, we'll just put it in the unprioritized backlog and we'll revisit whenever.

C

I also just add there is a workaround for this right now, so you can use a yaml fragment rather than a starlight function, to preserve white space in multi-line strings. It's not as elegant as this. um Basically because you have to use the ammo fragment, you have to assign that multi-line string to a key.

C

So then, when you use that yamo fragment later, if you just want to get the string you have to index by that key, I have an example somewhere in the public slack channel I'll try to link.

B

And I think this was brought up by um one of the community members correct carrie. I think that's how it kind of resurfaced: yeah, filippo,.

C

um I found a link here I'll, send it through the chat.

C

And you can see the example of the work around for that solution as well, that slack thread.

C

So um I have thoughts on this like there is a workaround for this solution, but the user experience is, uh I think, degraded, so it's not as easy as if we did support starlark multiline strings.

C

So for that reason I think that there's a pretty you know a pretty good reason to tackle this and try to get this feature back.

D

Yeah carrie, I would.

E

D

um It seems like the user experience is not what's expected when we say that we're using starlark- and I think that's a pretty big deal- that the workaround seems um a little more difficult to understand than just the normal uh function or multi-line string defined in the function.

D

So I'd agree that this I feel like this issue should be.

B

B

Do we have some kind of a solution for this, or do we need to still kind of figure out how to solve.

B

C

The issue that this originated from has some considerations for, like alternative approaches, how we can still keep or still prevent, uh starlark indentation error and keep starlock multiline strings.

C

So there may be some more detail in that issue that will like provide like implementation. Considerations like you know how difficult this will be, and that sort of thing.

C

um I worked on that issue originally, and I do remember that it's.

C

C

Not exactly easy to kind of know what the implementation implementation implications of changing the starlark parser is right off the bat, since I think most of our time, working on code and ytt is is definitely outside of that that starlark space um so yeah. I can't really say, but there may be some more detail on that issue. I'll try to find that and put that in the.

B

B

Other thoughts from other maintainers.

B

I could go ahead and uh yeah put this in the higher priority column in the unprioritized backlog, steven you're, going to say something. Sorry.

E

I did just want to know one thing that the uh in the two either cases there that either indentation is preserved or the limitation is clearly documented.

E

We have a sort of tucked away note in the ytt docs about the language that it is not white space sensitive and to me that sort of implies that uh if you have a multi-line string, we're not going to be preserving that white space.

E

So I guess I'm just trying to note that it's sort of already done. We did take care of that either.

A

A

Okay, um so helen, does it sound, like you, have an action item regarding this to move to a higher.

B

Priority or yeah I'll move it accordingly um and then wait for. I guess some of the details that kerry would add to this issue: um yeah. Okay, thanks for y'all feedback. Here, thanks.

A

Okay, moving on uh we have should we support regex matching for images.

A

Who put this in there? Do you want to speak more to it?.

C

Yeah put this down, I thought it'd be helpful to have a conversation about possible approaches for this.

C

So um this issue details a use case for k-build. um The user wants to resolve an image reference to its digest form and that image reference lives in a string so I'll annotate. Here the image reference is prefixed by some image, arg and so normally k build, can detect image.

C

References based on this string starts with the registry that the image is in cable also provides a way to match images based on their key um or other, like specific configuration that you can provide to k-build, but it doesn't currently support any type of regex matching that would be needed in order to match on this field. That has a prefix of a string.

C

um So there were a couple of suggestions on how this can be done. um If you scroll down a little bit.

C

um I think it's like the third or fourth comments from dennis.

C

Dennis suggested that you can use a environment variable to reference this image, which then, when it this uh deployment runs, it will replace that environment variable with your image name, so that way, cable is able to detect. Based on this key, that's named value that this is an image reference and cable should resolve it to its digest form.

C

However, the user pointed out that sometimes people don't have control over their configuration.

C

So the configuration author wrote their template this way configuration consumer doesn't have the ability to modify that into environment variable so providing that that functionality to match based on a string was kind of needed. In this case,.

C

um There were a couple other suggestions thrown out there that are really cool. Like there's this idea, I think at the bottom, suggested by dimitri of like a solution that uses ytt, so you can modify this configuration using ytt to use a similar format that dennis suggested, if you don't have control over the configuration.

C

But my question is basically.

C

What would it look like and is it a good idea to just add a regex matcher in k, build that would support this use.

F

F

This is the same thing that was talked about in one of the previous community meetings or office hours like has this. This ytt function been something that's come up before. I think I remember something about that.

C

Yeah it has, I can't remember the specific issue that it was brought up in, but it's like it is a topic that I've heard throughout multiple of our tools of do. We want to add some type of templating engine to other tools.

F

Where is that one cap, I forget.

F

Was it cap weight rules or like.

C

Yeah, um it was cap.

D

Yeah, I don't remember the specifics of that cap issue that we were suggesting this fix for, um but it was a similar kind of idea where it was a break. The glass feature of something that not a lot of people do. But if you are doing it, you were confident that you're, confident in what you're doing and so kind of giving you the freedom and the autonomy to do what you need to do with this feature.

D

That's really powerful that ytt templating or some kind of you know regex that we could accept the ytt function for there.

F

Yeah, I wonder if it's like a, I guess the specific ask here is about regex's, but, like I wonder, if there's some some standard way to enable this across tools, if it's something that keeps coming up that talking about this in conjunction with the cap issue, might make sense.

F

If we decide to go like the ytt function route, instead of um just some simple regex or something like that,.

C

Yeah, I think that makes a lot of sense like if we're going to add a similar functionality to cap. Then we should also think of this in as a future possible for k build if I were correctly based on what geared said like the feature in cap is sort of a more advanced feature that we were talking about.

C

um So if you need some sort of ytt templating in cap you're doing something that is a more advanced use case. For this example, this seems less advanced. We pretty much just want to match on a string.

C

So yes, ytt has a regex library that would be helpful here, but maybe it's more heavyweight than we need like cabled. Als already has um configuration that can be used to change how it matches images.

C

So maybe we can just extend that rather than add something as heavyweight as templating.

F

How does that currently work right now, like? Is it just um like it works by looking at keys and stuff.

C

um I posted a link in the chat uh for the documentation on how cable currently matches keys um nancy. If you want to open that up, it might be helpful to kind of view that all together.

C

This right here, I posted a link in the chat.

C

Thank you yeah. It's at the very bottom of that page matching images, so you can provide a cable. Config file has kind config and if you add these fields, you can specify how k build looks for image.

C

C

C

I think this is the right documentation.

C

Sources, image destinations and image overrides.

C

um I guess this is the default configuration what we're looking at right now.

C

So I'm not exactly sure what k build supports today. I think it's in these search rules.

F

So yeah so there's like a key matcher which specifies like the key to look at and then a value matcher, which says like an exact image, name and image repo and I'm assuming the regex, is just the suggestion of adding like a via regex. Here.

F

um But I guess this is the issue about one specifically in a in an.

F

F

C

Yeah, I don't think that the fact that the image reference is in an array makes this more complex. I believe cable already knows how to.

F

C

F

There's a section for search rules in this doc. I don't know under the schema, which I think is more details more of how to do it or like how to implement search rules.

C

Yeah nancy, if you can search this page for search rules, I think you'll find it it's also under.

A

C

um I think it might be down. Oh it's one word.

C

C

Is this correct yeah, I think that's an example of it.

C

um But the actual reference is under um schema, so there's a section I think up further in the stock yeah yeah yeah under schema. There uh just go down just a tiny bit right there yep thank.

C

You yeah, so these are the the options for currently uh configuring how you can match on image names.

C

So key matcher value, matcher, update strategy.

C

We could add one that is regex matcher or something like that.

F

Would it just be something under the value matcher instead of image and image repo, you also have like a regex or something. So if a map item or index has a value that matches that regex, it selects it.

C

That's a good idea: yeah.

C

So I'll take notes of this conversation and propose that in this issue, um if people are curious or have thoughts, um love to carry on this conversation in the issue and get to a point where um yeah we feel like, we can propose uh some solution to this.

C

A

Okay, thanks carrie, um that is the last item we have for discussion topics. uh Is there anything that anyone else has as far as discussion topics, or does anyone want to add something here to any help needed?

A

I saw that we have pete on on uh joined us pete. I believe you you work for.

G

Vmware uh yeah, I do.

G

um I just wanted to join just kind of out of curiosity and hear the process, but also um a day or two ago I posted a question in the uh the the vmware slack, um there's a project that I'm working on, uh involving taking a helm chart and trying to find, like all the possible permutations of uh image, references uh for the for the purpose of trying to rewrite them for like uh like seeing if we can find all of the image references not just like the default ones, but and for any values in like the values ammo to migrate, all of those references or all those images into a private registry.

G

um So I just kind of threw that out there and trying to see if, if the uh the tools that you all provide would would work for that, I got some links and honestly, I haven't gone through and read all the details yet, but just want to throw that out. There.

C

I haven't seen the slack conversation, I don't think, um but do I understand the problem correctly where you say you have a helm template and it has image references in it. You want to find all of those image, references and collect them.

G

So yeah I mean like one of the things with helm is that you know you could easily have you know it's like a mysql. You could have like my sequel, colon and then image, and then it's like the full image reference with the registry, the repository and the version.

E

G

Like that or you could have it split up so it's you know, registry, colon, something and the image is something else. There's so many different ways that you could like reference, an individual image. So I'm trying to find a way to to extract all of the images that could be used by that helm, chart into a list and then the second half of the project is to rewrite either directly rewrite the helm, chart or the the values or something like that, so that it can update to a different registry.

F

Do you have like.

G

An example home chart uh not directly at this moment.

F

G

Early in the project, so yeah yeah, that makes sense.

G

The one idea that we had was to uh ask the author of the helm chart to provide kind of like a like a search path, type query to say: okay, here's how you can you know executing this query against your chart will give us at least the path to the the images that you've defined, uh and then we can use that as a hint to to know how to rewrite them smart.

G

But I was just kind of curious to see if there's any tooling out there that exists to define that without asking for you know, modified telemet charts to ask without specifying like that, the helm charts need to be written in a certain format that we can kind of take anything from the community um or uh or asking for extra work on the part of the of the helm chart author.

F

Yeah, I think, like k-build, sort of what we were just looking at is probably the most helpful solution here that we've got.

F

um I think some of the cases you were mentioning might be a little more difficult or not able to be handled by k-build, and I think another aspect that could be tough is like you're, saying, collecting all the images and not just one permutation of the possible ones, but because I'm not sure that k build can take in like raw helm templates, because I don't think they're valid ammo, um oh yeah, fair enough, which means that, like it, would only be able to work on rendered templates and in order to render them you'd have to sort of switch.

F

The values around to get all different possible renders so right. It might be a little limited in that sense, as well.

F

That's interesting.

G

um Yeah I'll I'll still poke around with these tools, because I think that there's a lot of good value there um yeah uh part of it was you know, I I see the work that y'all are doing and I'm I'm interested in it and I'd love to to see if it if it matches my use case too so.

F

Yeah, if they were ytt templates, I think k build, could do it. Yeah.

C

Yeah I see in that thread, um natasha chimes in. um I know that she is doing something very similar and was able to use the post render feature of help of helm to um basically resolve image references.

C

I think that's because, like during the post, render stage output valid yaml and then cable is able to find those image, references and update them. So it's yeah. She suggested that that's definitely something I'd look into.

F

A

Okay, anything further on this topic.

A

E

A

Anything else that anyone wants to bring up before we the end the office hours.

A

Go on once twice all right, uh so yeah! Thank you. Everyone for joining today, um some really great discussion to be had um if you're watching this from home and would like to join us for uh another meeting or office hours that we have. We do have our carville community meetings every monday at 11, 30 a.m, pacific time, 2, 30, p.m.

A

Eastern time, if you can't make that we also have these office hours that happen every second and fourth thursday of the month at 11, 30 a.m, pacific time, 2 30 p.m, eastern time- and we hope that we do see you at the next meeting that thank you, bye.