VMware Carvel Community Meetings, 22 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Carvel Community Meeting - February 22, 2021

Description

Carvel Community Meeting - February 22, 2021

Topics discussed include "why imgpkg is inefficient," "kapp should double check that K8s API returns labeled resources," and ytt schemas. Cari, a Carvel maintainer, also covers what is "pointing" and why the team does this.

More details here: https://hackmd.io/F7g3RT2hR3OcIh-Iznk2hw?view

A

All right welcome to carmel community meeting. uh This meeting is being recorded. Just a reminder to please adhere to the code of conduct which is listed in the agenda.

A

I will post the agenda in zoom, so everyone has it handy.

A

And we will kick off today's meeting with aaron he's going to be going over some of the backlog items and we'll work through that and then go on to discussion topics so I'll. Let aaron take it from here.

B

All right, thank you. Nancy share my screen.

B

Alright, so as some brief status updates, um we do have the ytt schemas track uh still in progress. um Maybe I'll just bring up the zenhub board. Now, let's collapse everything else for now, um so we do have ytt schemas still in progress.

B

uh We have here yeah a story that we'll be discussing today um and then, following that, we just have a a couple other stories for for documentation and do some usability testing. um So that's what's remaining on our schema mvp.

B

Other things that have been in flight we have. This include non-distributable layers work on image package.

B

We currently have a final story in flight, and this is just about the the uh go container registry um pulling well. They have already emerged in our changes or upstream contributions, uh we're just waiting for a new release to be cut so that we can consume those contributions.

B

So that's the only thing remaining on that track of work. We do have um some performance related work uh for image package that we currently have in flight still very much in flight, but we have found some some interesting findings so far I don't know dennis or carrie. Anyone want to provide a brief update on the story.

C

Yeah, I mean we did a bit of profiling and we found a few different ways.

C

We can improve things like not calling certain things twice when we can only just call it once and that still works and instead of calling a get call of a head request, because we only want to digest so we just have to download all that blob content to figure that out and also we're gonna, also, if, if possible, mount blobs if you're copying from the same registry- and so you don't have to copy the bits- you just essentially create a sim link, and this table just shows the performance improvement.

C

Really so like. Like the second row, we had a docker with 20 images, but there's only one image: 20 times it's a gigabyte, large and the old way used to take 14 minutes. Now it takes like just under 8 minutes so and the only and the other nice improvement was the light. The second last one local registry, again 20 images which only one distinct and it's a large image. It's a gig. It used to take six minutes now.

C

It only takes one minute: um that's a local registry to local registry, so in theory, we're going to see nice improvements for, like large images, really that most of that came from um mounting the blob and we're also going to leverage one more thing, which is when we write multiple images. uh We're gonna leverage like this go container registry multi write function, which has performance reasons why you should call it the biggest one. Is it dedicates all the layers?

C

So if you have like 20 images and there's like 10 layers in common, it will only upload that one layer twice for you um still in flight.

C

um This is the remaining tasks left to do on this on the story.

B

D

Hey aaron, while we got this one open, I wanted to pass along observation uh that dmitry made, and I thought this was was helpful um in addition to like the well-ordered uh cases, might be handy to grab one where there's decidedly a distribution of shapes of images and size of images. One such project is um see for kate's. I think it has it. Last I looked.

D

There were like 49 images in the whole product, um so being able to get a read on that as well might might be helpful because sometimes uh performance characteristics shift depending on you know when there's a greater degree of variance, um so that might be signal just just kind of passing that along.

C

Yeah we can do that, although all this work is about doing less calls and less uploads. So I'm happy to do that, but I would continue down this track and then once we have a final kind of stable fix, we can check out the old branch run it with cfocate and then check out the new branch and I think, we're going to see a performance improvement still yeah nice.

B

Cool sounds like a plan.

B

All right um and then looking at the rest of these, I don't think, there's anything else that we need to touch on. um So I think we can get started then into what are this week's stories that uh we should discuss and point.

B

So we have a a new uh prioritized epic, which is a recursive relocation of recursive bundles within uh image package. um Would you want to speak to these stories.

E

Yeah, I can do that so um so before before we dive into the the the stories in themselves.

E

Maybe it makes sense for us to talk a little bit about what this is um so like this feature, the idea behind this feature is image. Package is able to create a bundle with your configuration plus images. It's also able to copy that bundle from one from one repo from a registering repo to another one and you're able then to pull it into execute uh and install and deploy that bundle in itself.

E

So this this functionality already exists, but there was a caveat that never we never implemented or that currently is an error. That is what happens if your bundle, if you're, trying to bundle a bundle so current. At this point, we only had bundles of images. They are like separate images, and what this proposal brings to to table is the ability for our packager to package separate, smaller bundles and then have like a bigger bundle that they can bundle all these things together.

E

So if we think about any a good example for this could be uh cf4 cates, where each individual component could bundle its own piece and then vmware as a whole could just pick it up and just pick up all those bundles together and just bundle them back again, so they can distribute as a whole.

E

So this is more or less what we're talking about in this feature and in order to allow this workflow to work with image package, we need this couple of stories that are here. So, let's start with the simplest story, and that simple story is basically allowing our users to create a bundle that have other bundles on it.

E

So if we follow through the the acceptance criteria, uh we have, I believe, two scenarios on this on this story. The first scenario is that the user can create a recursive bundle that a bundle that contains another bundle. So if you see there, we have a little bit of of our images lock, and so, if the new bundle that we're going to create points to another bundle, if we do an image push of this folder, then we should see nothing different, but we should see the this working.

E

So currently, if you do the command the image package push minus b, with a bundle that contains other bundles. It will fail. So this is the the. What we're trying to achieve here is the ability to okay, let's remove that card there, and then what we want to see as well is that when you do a pull of it, it just nothing changes, but you can still pull the the bundle.

E

So this is the first uh scenario that we have here. The second scenario is exactly the same scenario, but it has just a little tweak. That is, we have a bundle that contains a bundle and an image, it's just to make sure that we do not have any regression and you can still package images with with our bundle. So those are like the two scenarios that we have here.

E

uh There's a note there on the bottom that says that the image pool um will change the the output that you see will change in the next story in the future, but for now this in this first story the pool is going to look exactly the same. There would be no difference so does.

A

Anyone have any.

D

Questions when you're saying that the output of the image pool wouldn't change like what? What is it that we're not seeing.

E

Just it's just like a reference. It's there. It's like nothing, will change. That's! That's it! That's what it.

A

E

If, when we're gonna see the next story, the next story is going to change the way pool looks like, and if you, if, if you saw the the the proposal, the proposal, I had like a very different output for pool, and this is just a call out saying that we know that this change is going to come up. But it's not in this story. So this story, we expect nothing to change at the pool level.

D

I see so that the reference to a bundle exists, but poole doesn't respect it yet.

E

C

So after this story is done, we have scenario one and scenario, two committed pushed merged. How does copy work like if we copy this bundle to a table? What would the output look like over the command exit with.

E

At this point, like, unfortunately, that it's very hard to create a story- that's small enough that contains the full breadth of this change. This change- it doesn't make sense until you get to the story that allows you to copy- and that's only the fifth or sixth story in this group of stories, and the split was more function. Wise, just try to get smaller stories to get us along the way, and this will it will become a strange uh position if you try to execute this with bundles that have other bundles.

E

Until you get to that last story,.

C

Okay, so it just won't work for them. Don't add any checks in the copy code to say you know if you're copying, from a bundle to a tough I'll like give back a warning. Just like don't do anything, don't make any changes in copy. Basically,.

E

uh Yes, I don't, I don't think we need to do them right now.

E

We could, if we think that's that's a a better thing or it could even like what's going to happen right now, if you keep, the copy is that it's only going to copy the the bundle image so in the end, you'd still have like the bundle image with you, but the rest would be in the other registry, so there would be like a breakage.

B

Here something to take note of, as I mentioned, is it was kind of tough to, I think, find the appropriate slices for these stories and they might not be completely exclusive of each other or in fact they are likely not um if it if it makes sense like. Maybe I don't know you can always have more than one of these in flight or if we want to reshape these stories in some way, I think that's all all open uh for feedback.

B

We want to make sure that, like you know, the way they're shaped is helpful for the team, but these are also a tricky set of stories to write in that standard. Like you know, super thinly sliced way.

D

One way I think about these things is, we have a one part of the workflow is high fidelity support. That is here, creating the bundle and then it drops off where it's like. Well, you need to know that you have a bundle that refers to a bundle and you've got to do that additional copy yourself in order to make that work, um and then that way you can sort of regard the whole flow as complete, but degradated fidelity of implementation until we get to those later stages.

D

That's one way, I'm able to rationalize these things, but that does bring up a question in my mind um about how we sort of draw the transaction boundary, and I don't I just mean that, like as it as like implementing a full feature around this whole thing, one one way we've done in the past. We put a feature flag.

D

Another way we can do it is where, with with each story, we're keeping documentation- and I think then dennis you brought up the point about- maybe a warning: um keep that up to date so that uh we can continue to always have a releasable product and the experience that the user will have um is at least honest uh to say, hey this isn't this is going to completely happen the way you expect it. It will be soon or something like that.

D

um So so then I would propose. I would be thinking about this when I'm pointing this story, I said we need to do a little tiny, something to at least emit a warning for pull and copy.

D

D

F

I have a clarifying question.

F

So so, to make sure that I understand that the note at the bottom um I'm a little confused for making a distinction between the output of image package poll that you see in the terminal, whether that is what doesn't change or whether the um like output directory, that you specify that doesn't change and like whether you would see the recursive bundle. Contents in that output directory.

F

As part of the story.

E

Okay, so I see that I shouldn't have put that note there. This note was here just so like.

E

If you follow the proposal, you see that the output, what you see in the screen, it's completely different from what is you're going to see what you see right. Now, it's not going to be even remotely similar to what you have in the future, and that was the note just to clarify that the behavior will change and what you see and what will happen will it will change but not right now. So that's what that's this that's what this node is talking about. So if it makes more confusion, I can just remove it.

D

I I don't think the note itself is a problem, so I think, actually it's great, that you brought attention to this, so we can be clear together and what the story means.

F

Cool yeah, so basically, the output directory will not change as well as the output text, cool.

E

This is a change. This change here is a change on the push side of the world.

E

That's basically what's going to happen here uh about what you're saying john, we could add more things, I don't we can talk about it, but we should define it on the story if there's something that we should add here like if you want to add some some messages and so on, then we should add that information here, but that will link that will increase the scope of work of this particular story, and what I was going to say is that I am perfectly okay of having a head of develop.

E

That's not 100 working for all the use cases for the time being of this feature, but at least that's my uh my view.

C

Would it be releasable like it might not be working for the use cases, but do you think it's releasable if we needed a ship like a security fixed, would we have to do some workarounds or is it just commit and develop and then release it as well.

E

C

E

Going to work like there's, no there's no, this will not change anything it. The only difference is that previously we got an error saying that you couldn't push a bundle that contain other bundles and now you'd be able to so that that's the only difference of from what currently exists right.

C

Okay and one last question- and it might it's probably answered in a future story, but there's no validation required like it's. Just the image key and an image might be a typical image or might be a bundle, and we don't have to look into this bundle to make sure that there's no circular dependencies or whether it's a valid bundle. It's just it's just text for the time being. That's part of the story like no no, no image introspection or anything like that required. There's.

E

No need the push. The only the only thing that push does is just collect. The the images creates a layer and push the pushes the layer, so there's no validation. That needs to happen at this point. Okay, anything's. There thanks.

D

Yeah, the only.

E

D

Remaining, then, is the product question of. Is the user experience of doing a pull on? What I thought would give me, recursive bundles, doesn't and currently would be it would it would just not do it silently? Is that.

C

Okay, I think it would be confusing if we shipped it like that like if like, if we do the story and then we release it and people use it, I think they'll be confused.

E

Maybe I'd be okay, adding a feature flag to this pool for now, for the time being and then we can remove it.

B

B

Okay, so it sounds like we want to address it I'll, leave it to the individuals that pick this up as to what way we do that, uh how about we do a round of pointing to see where folks are thinking right now,.

B

Cool ready to point: let's go three, two one shoot.

B

All right we got some ones. We got some twos uh how about here from the.

E

Twos, uh I think this is a story. It's a one story, I'm just adding like a little bit of uncertainty like if you want to do the the flag. If you don't want to do the flag, if you want to do like anything else, extra to just communicate to the user, but in the end I think this is a one point story.

B

Anyone from the ones want to speak.

C

I put a one, it sounds like we're: removing a guard and probably adding a feature flag. It didn't seem like a lot of complexity or I didn't think I can't think of too many unknowns either.

B

All right, let's try pointing once more see if any, if we develop further consensus, either way. So three two one shoot.

B

All right, the ones have.

B

It shall we go to the second story.

E

Okay, so this story contain this story is like the second step that will finish the first complete flow of uh of image package, and that is someone picks up. A bundle pushes that bundle to a registry and someone somewhere else will pull that bundle, and it will do things with the bundle right so like this would be the first complete flow that image package has. So what this story has.

E

It introduces um a new behavior on pool where now, if you provide a flag that is minus r, what it will do is it is going to go through all the images that contain on our image um on our image.

E

Oh forgot that part that's awesome, or did I not? I think, is it there we'll see uh so like what this does is it will pull all you to pull the image plus it will pull all the bundles.

E

It will pull all the images of the bundles that exist inside that bundle, and this could be recursive. This could go way through like 10 deep, and the first thing that we'll see is that the output now of our pull minus r is completely different, and now it has some sort of like indentation and we can see like if this is like the best.

E

If there's a like anything that could make it easier to read or not, or we can just postpone this and see what the users give us some feedback on, but now we have like some structure and we're telling it there are nested bundles and that our bundles do have nested things inside of them.

E

So that's our first uh scenario. If we go to our first second scenario, we see that this is also a uh yeah. I think I forgot.

E

Oh that's sad, okay, so I'm missing a scenario there. uh So this is just to say that there's like a minus minus recursive and it's missing the the more important part of this feature, which is kind of sad. But let's go to the third one and I'll talk about the second one. The third one is that if you do a pool without a minus r, it will do nothing, but it will print some more information about the nested bundles okay.

E

So uh the scenario that we are missing here is: that's all fine and good, but we need to pull the folders and that's the part that is missing here. But if you go, if you scroll up, does it have a link to the proposal.

E

It does not, can you go to the proposal.

E

E

Okay, if you scroll down to the pool area.

E

Okay, there we go so when you do a pool minus r. uh This is our expectation. So if you do a pull minus r, all the bundle configuration all the configurations for all the bundles will be placed inside a folder called bundles inside our image package, particular folder and they will be they'll. Have the sha. The the name of the folder that contains the configuration will be sha dash in the sha of the bundle.

D

This itself is not nested it's flat, regardless of how many layers of nesting there is in this tree of bundles. These will all be flattened to just one huge thing of of shaw named directories. Exactly okay,.

E

So I am so sorry that this didn't go through.

E

uh Does it make sense.

F

So how is that different from dash dash recursive again.

E

So this is what what is expected to happen when you do dash r or my dash dash recursive.

F

Oh okay, they're the same com. It's.

E

The same it's as short, the short flag and the long flag. That's it's exactly.

F

Okay, the behavior is exactly the same.

E

D

Okay, so if, if I'm understanding this right, joel, the only missing piece was just and if you look on disk here's what this looks like exactly.

E

Okay, it's missing just the output of a tree in in one of the okay in one of the criteria- and I thought I did have that.

E

On the second scenario, but.

D

Apparently I did not so do you intend it was there? Is there any other difference between scenario, one and two other than ones using a short form flag? Okay, so there? The behavior is the same. It's that the input is either short or long flag. Yeah.

D

D

I assume we want to say something to the user if they're like a a warning of some kind, I don't know like some sufficiently loud signal that this thing contains additional bundles which were not pulled. If you need them, please use the r.

D

Is that covered here or is there some subsequent story that will enhance this.

E

uh No, we did not like the assumption that that piece that you see there it would be enough, but if y'all think it's not enough, we can add some sort of like text. Maybe here as it is part of it, that would say something like if you want to get all the bundle information just choose the minus r.

D

Yeah, I I don't read from what I see here, heads or tails, whether or not it is or isn't included. Personally.

F

Yeah, I agree. I think that makes sense. We should. We should add that warning and, alternatively, if you use the dash r flag and your bundle doesn't contain any recursive bundles, that should just work as ex as expected as normal, and it will ignore the flag.

G

Yes, I would expect that to work as normal, but provide some sort of warning about hey. You didn't actually need this. There were no recursive bundles.

F

I I don't know if a warning is necessary. I can imagine like this command being used in maybe like a ci to work, no matter what whether a recursive bundle is specified or not, it just always will work.

F

Curious differently.

C

Yeah, maybe you've gone through it, but why do we need to dash that recursive flag? Why not just always recursively get all the bundles like why yeah.

E

E

This was the the original like idea for the proposal. It would just always get it, but the use case that we currently have for image package is the usage of it being it's being used on the cap controller context, and the cap controller does not care about the bundles, the recurs, the bundles that are inside the big bundle. So this is going to be used as a in the cab controller world is going to be a package, and a package contains contains pointers to other packages and information about each one of these packages.

E

And when I say packages, the term is interchangeable with bundle in image package world, not at cap controller level, but at the image package level. It is a bundle, so the use case that we're targeting for now is it it's possible to just download the configuration for this particular bundle, and this would be the main use case for image package. But we will also allow for people to download all the configuration for all the bundles using the minus minus minus regards to flag.

E

Did the explanation make sense.

C

I have enough information to point.

B

Okay, how about we do around appointing that.

F

We um done the spiel on on why we point yet, since we started this new format.

F

um I can give that feel now of like what what pointing is and why we do it.

F

um So, basically, we provide a estimate of complexity on stories, just the engineering team and this complexity estimate is based on stories that we've completed in the past. It's all relative across all of the repos that we maintain, and so um we count to three and throw a number of complexity on the fibonacci scale of how complex we think the story is um that complexity estimate goes into calculating like velocity and can help to be used to track, like uh estimated timelines of when work will be done.

B

B

Cool all right, let's, let's demo, what pointing looks like.

B

All right, three, two one shoot.

B

All right, we got some threes and we got some twos.

B

Let's uh start with.

C

C

Well, I pointed at three um I think, yeah just test driving the changes to recursively get bundles.

C

I don't know it just feels like a like a big it sounds like first step is: usually we have to build all the testing infrastructure to do this so, like I feel like we're going to do a little. You know adding test to begin with. Also recursion can be tricky and making sure we don't have any like you know, infinite loops, or we deal with the cyclic dependency issues and making sure that all that all works. I also have questions about compatibility.

C

Moving forward like you know, bundles, have images, locked images, yaml and I'm gonna at least ask some questions about how we future proof this a little bit. Not maybe it's part of the story, but just thinking about it like there's a few like on you know, low level implementation details that I like to think about as well. When doing this recursive bundle story, I don't know.

B

How about hearing from a.

E

E

I am a high too so.

B

Okay, maybe there's not a lot of opposing perspective, then uh let's try pointing once again just to see where we're landing then so three two one shoot.

B

All right looks like threes, we'll take it.

B

Okay, let's see time check, we got a lot other things to get through. Still, let's see- and there is a discussion topic on testing frameworks. So, let's see if we can get through.

B

um A cap story, at least so we are prioritizing some cap work. uh Okay, there's the rest of it uh this week. So there's a couple of issues that um I don't know. I have uh we've heard from a number of folks saying that this is you know something: that's that's blocking some of their adoption of cap. uh The first one we'll talk about here is cap, should double check that kate's api returns, labeled resources.

B

So right now, I cap expects the communities api to return. Resources based on the label. Query specified.

B

As dmitry mentioned here, when folks are using the aggregated api implementation, there is no guarantee that resources are filtered by the label, as cap expects uh andrea being one of the examples being called out here.

B

So this story is determining a way to to work with these types of resources. Resources that may not have the uh may not be filtered by by label as uh cap would expect from a kate's api response.

B

um Some details in here I see daniel asks a question about like what would be the behavior expected here and dmitri responded that it would eliminate the resource from further processing.

B

So it would allow cap to keep working with these types of resources. It just wouldn't do further things like diffing.

B

uh Do folks have questions or does anyone else want to add more context to this issue?.

C

So in the description it says uh it appears that if aggregated api implementation come into play, what's an aggregated api implementation, I think I missed that.

D

Yeah, there's a bunch of machinery being referenced here directly in kubernetes, there's a way to extend the standard api to actually add your own machinery. To that you augment it, but it appears as if it's just the kubernetes api, you don't have to do anything new to uh to reference it to use it as a as a client. So it's a way of being able to um to augment that, and tools like andrea will do that they might do that because they want to add additional um processing around those kinds of requests.

D

So it's kind of an advanced extension point for cluster machinery and it's their job. They now have like a whole set of responsibilities that you get out of the box if you stay within the kubernetes resource api.

D

One of them is that when a query is sent, if it includes a filter that says, don't only give me resources that have these labels. You got to make sure that you only return those resources with those labels.

D

Now some folks haven't done all the work and they return back resources that don't have those labels and then caps like this doesn't make sense.

C

Okay and this aggregated import api that you just discussed, it's not experimental and it might come into play like this is something that's already out in the play. So it's like it's not if it appears if it comes into play. It's like it's probably going to happen and we do want to handle this. Like there's no point.

D

C

And waiting to see if this is adopted or not, okay, no.

D

This is just about in a particular operator's life that, like they might come into third-party software, that will do this yeah. That's they come into play. Yeah cool.

G

What's the current behavior, if we get back way more than we expected when requesting these labeled resources.

D

D

So what the one way I've thought about this is what we're doing is we're trying to help the operator in a situation where the third party extension on top of kubernetes, that they're using isn't behaving to spec.

D

So, instead of just like stopping completely, there may be little things we can do to make that experience smoother. So here's one where we get back resources- and it's like oh we but the aggregated api intended to do- was only provide these resources. So cap is accommodating that it should have been done to spec upstream, it's not but cap's gonna correct. It.

D

Oh there's another piece: sorry context-wise uh uh paul dropped a uh a pr there and this was meant to be a draft pr to help him work around um not being blocked, uh be inspired by that. But you, if you dig into the pr you'll, see that there's a um a few tweaks that are required um if that's exactly the right, but it's a nice expression of like um a short version of here. Here's basically the behavior that needs to change. So that's pretty nice! You get a nice line up there.

C

And so like they have an aggregated api input version, one it doesn't implement labeling or filtering correctly. We ignore it. I think cap ignores it and then you know aggregate api input version two comes in and gets installed and that does implement filtering correctly.

C

It's the behavior here that cap would just see that as a brand new resource, like I'm gonna what you know what's the behavior around there.

D

So yeah so that first time around with the with the cap, fixing basically fronting for that aggregate api's mistake was that those resources don't get included, it's as if they never got returned in the first place from the aggregated api, so that so what gets actually applied is the proper set or what gets incorporated into the diff is the proper set of resources.

D

Then, when the aggregated api respects the label filter cap just doesn't have any more work to do. There's no filtering.

C

That needed to be it should like a no diff. It's like it's already been applied correctly, assuming there's no.

D

C

The source code in the source file.

D

C

E

It will still use the extra cycles, though, to do the filtering, because the filtering still needs to happen, but because you don't know which which like that, if the resources are being retrieved from uh one api or not or the other, so you'll always have the cycles. Even when andrea and andre already fixed this problem on the latest versions, but uh we will always have to pay that costs for them, not having that from the start.

D

Yeah, I have an assumption that is incredibly cheap compared to all the other operations happening. Is that a good assumption.

D

I I believe what we're talking about is looping over some objects in memory and dropping them. If they don't meet a criteria, it depends on the amount of objects. But yes,.

E

But it would be more expensive, like it would be less expensive if they just returned the amount, because you could receive like 10 000 objects, where you just care about two so like the expense would be in other places, but yeah.

B

So, maybe just something I'll note is: we are kind of just picking up this issue, as is so there is, you know, potentially more room for, like definition here I would say for the folks that pick this up feel free to you know, share around what what you're planning to do once you kind of dive into this bit more and get get feedback.

B

uh You know feedback sooner than later, just just to make sure the team's in sync on it cool how about we point to see where people are landing right now, three, two one shoot.

B

We got some ones, we got some twos uh either side want to speak to their stance.

H

Yeah, I saw this as a as a small two. um I originally saw it as a one, but then, when you mentioned that there might be some investigation and like some story writing and some feedback with the team about like what exactly we plan on doing here that bumped it up to a two. um The technical complexity to me is not very high, though, and I feel like that sentiment might be shared among the team.

G

I'm not sure I agree with that. I also threw a two, but uh I guess for a different reason. The reason I gave it more than a one is that I think I'm being asked to implement a kubernetes labeled resource selector, and I just don't know what goes into that.

G

So for me, that's much more uncertainty than expectation that this won't be that hard of a problem.

D

It occurred to me that maybe there's a tiny bit of documentation here too.

F

I pointed as a one because.

F

To me, this seems like just filtering resources that were returned based on a single field, um and I don't see much technical complexity.

B

There all right, let's try pointing once more see where folks are landing. Now it's three two one.

B

Shoot all right, I think, twos are gonna. Take it.

B

Okay, so let's see we are short on time. I think we wanted to get through a schema story still.

B

B

And I also know we have a discussion topic.

B

I'm curious the urgency of this discussion topic.

D

E

Okay, uh I think it's going to be quick, but I don't know if we have time, but in the end, like my objective, is we we voted on what we wanted and I just wanted to. Okay, so shall we start doing this right now or.

D

E

That's that that would be like what my expectation for the discussion. This topic is.

D

Did you want to like maybe include this in in tech forums, and we can give it like due time to like make sure that we're all cool with what that means and everything.

E

D

We have at least one tech forum topic this week, but we can make sure that we make space for this yeah appreciate you moving that forward.

B

Cool, thank you all right. So then do we have enough time to dive into this ytt story. Let's give it a shot.

D

All right all right.

D

Okay, so private libraries right so when you invoke ytt you've got you're, basically running a library. If you will like a ytt library, but then it could have other libraries that are defined under that underscore ytt lib directory and a core feature of that is that they feel like they run in their own space. They have their own templates, their own data values and, with this story, their own schema, and it should have that sort of recursive feel in a sense that that schema is applicable just to that library.

D

And so this story is an attempt to try and describe what does that mean? What does that look like.

D

So this first scenario, so let me give you a second to read this over.

D

So this is happy path.

D

Any questions about this one, thank you carrie. um So what I mean by private library is um in this example phuoe. I think you just want me to say the word phooey phooey is a private library as opposed to what well, if you create a um as opposed to um uh jeez, okay. So so I guess I guess this is a little bit of a technical like an implementation detail, but in ytt every directory is actually its own library um it that in practice turns out not to mean too terribly much.

D

But if you had a subdirectory here that was not under the underscore ytt lib, it's considered public directory and it's all sort of a part of its parent. But this underscore ytt lib says well. Anything under here is disconnected from the parent and it will be ignored in terms of uh execution of the parent, and the only way to include its output is to do something like this, where you programmatically pull that library in and here we're saying, give me the foui library and take its contents and evaluate it, and that returns a doc set.

D

And then we're saying here replace this document in the main library. With that docset.

D

Did I explain that.

D

Well: okay, any other questions about scenario, one thanks for asking that carrie okay. So let's take a look at scenario two and scrolling down here thanks, sir okay, so.

C

D

Is just like the one above it except now, we're gonna include a data value that doesn't match the schema and there's two ways that you can provide data values to a private library. One is I'll call. We call it declaratively where in a file, that's included in the initial invocation.

D

You include a yaml document, that is a data values and you direct that data value to the private library.

D

So this is a way of an end user, regardless of whether or not the author of that public library intended or not, you can reach in and say here tweak this piece of data value, this configuration for that private library phoey and set its data value to this value.

D

Unfortunately, foo is supposed to be a number, not a string. Thankfully schema catches that says hang on here: here's your output.

H

Hey john yeah, when errors happen inside of like the evaluation context of a library, does it usually output, an error that looks like this like during the evaluation of this library? This output happened. Okay,.

D

Yeah yeah great question I verified this is what this looks like this. This top portion is what it looks like, and so our job is to make sure that the reason is coming from schema.

E

Is that path, okay shouldn't it have like the full sorry, I'm looking at the last line of the output saying that schema.tml colon for it kind of feels like rule, we should have the rest of the of the path to get there.

D

The rest of the path.

E

So like it should be underscore ytt underscore leave slash fully, slash schema.yaml colon4. I see because you can have multiple. From now on, you can have multiple schema files.

D

So today we don't qualify in error messages. We don't qualify the the file name. I agree with you. I think that could be really confusing uh because you don't especially if you had two files of the same name, let's, um let's capture that as a a feature, enhancement or a bug it depending on your perspective, but we'll say that's out of scope for this story. Let's keep whatever the current behavior is for now.

D

Great question, though, I'll take note of that.

E

Also like do we need to have one data values, file per private library.

D

E

So you can have multiple ad library, slash refs at.

E

E

I I sorry I have. I have a I'm having like a very hard time here. What part of this is already implemented? What part is not because to? I? Don't.

D

I don't know private libraries gotcha so so schema as a feature is implemented uh type checking. What is not implemented is that we, when we go to do this library, get eval we're not incorporating schema checks. We're not looking for the schema within that library. We're not uh incorporating that check today, we're actually using any schema, we're actually saying, there's no schema um and then data values that are being set aren't um aren't being checked against that corresponding underlying schema.

D

E

So right now the the piping for passing the variables into the into the private library. All those things is already done. You unity. We don't need to do anything there. That's correct! Okay,.

D

Yeah great question and aaron, I realize we're at we're out of time here um I hoped at least we could sort of start loading up the idea. We could um do like a special, pointing session to come back and finish talking about this, instead of trying to rush through it or drag into folks next commitments. So, okay,.

B

That sounds good yeah. Maybe we'll find some time. Oh we'll find some time cool all right. Well, we covered quite a bit of ground today. um I guess nancy. Do you want to close us out.

A

Yeah um thanks everyone for the great discussion I feel like we got a lot covered, um even though there was a lot more to discuss for those watching the recording after um we've already had this meeting just a reminder. We do have these meetings every monday unless there's a holiday, so if you are able to, we would love to see. You join us next monday march, 1st for our community meeting.

A

Otherwise um you can join us on slack uh the kubernetes slack channel for karbal, and we look forward to engaging with you bye until next time.