Kubernetes SIG Apps, 4 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Apps 20200504

Description

Led by: Janet Kuo
Co-hosts(s): Adnan

Announcements:
- Working with KK from SIG Storage on StatefulSet Volume resizing

Demo:
- kpt [pwittrock, mortent] - https://googlecontainertools.github.io/kpt/

A

Okay, welcome everyone to May 4th 2020, see caps meeting and should I say: may the force be with you so today we have one item in the agenda.

A

We are going to do a demo of open source tool called caps. It's spelled like kpt and I'll, let Phil and Morton to talk about it.

A

Already I just.

B

Want announcement, real quick, oh sure, is that is a follow-up from police good, so it with the staple set by um resizing I talked to six storage. There's an implementer from sue storage that wants to move forward with modifications to the existing cap, go by KK, so I'm going to work with them and try to get that implemented and Anand. It expressed some interest in and try contributing to that as well. So I think we have enough.

A

Phil, are you there yeah.

C

I'm here all right should I get started. Yes,.

A

C

Right yeah, that's my dog!

C

C

So I'll share my screen here: post disable the tenon each screen.

C

You generally, you enable able to enable screen, sharing or something special on my going on mine.

C

D

Be easiest at Phil as a co-host because they change the zoom settings for all of the community meetings that only host and co-host Kate can share the screen and that usually works the simplest and the fastest. Okay.

A

I, add you as a co-host and Morton.

A

A

C

C

Are you able to see my screen yeah, okay, cool, so I was just gonna start out with a little bit of background information and then do a demo.

C

D

C

Demo, it's helpful to give the context of what we're trying to do and why it was designed the way it was, and one of the themes is about. Shifting the resource model left the kubernetes resource model. We often as we talked about it's often in the API server or in the control plane and is built using controllers and muting web hooks and validating web hooks and resources right, and these are kind of the primitives that allow systems to be built on top of the resource model, where the resource model are a set of declarative API.

C

Is that have different programs which read those api's and write to them and I'll lose this lily couple by a number of different api's together and what kept is doing is taking the same resource model and shifting it left, and so that means that hmm operating on the kubernetes resources can be done before they're applied to the cluster. Before you even talk to a cluster, for instance, and some of the reasons this shift list is useful, more abstractly.

C

You can think that the earlier you catch errors on in the system, the better the less likely they are to go out into into compacting production systems like an example, is compile time errors versus like runtime errors, if it's possible to catch an error at compile.

D

C

Would be better than then waiting until it's actually been pushed and then catching there, then so, the more we can shift the work left, the more opportunity we have to catch errors before they go out, and one thing I'd really like folks mmm watching this too, to kind of take note of is the architecture.

C

I think this is probably the most interesting topic of discussion, rather than what individual feature happens to exist today, because kept is still new and it's going to develop new features and probably evolve in different ways, but this architecture is really the foundation of what we're trying to do, and so the first three bullet points here that I think capture a lot of it.

C

The first is the output of kept should be kubernetes resources and maybe open api right, and so this follows the kubernetes resource model, where controllers, for instance, are writing to resources in the API server, and this basically means like you need to be able to COO control, apply whatever kept outputs or you should, or it should follow roughly like what the kubernetes examples on github show. It should look like these things. All right- and you can write these things to different locations right, so it can write resources to files. The standard outtake guide, endpoints.

C

These are all about locations. What's important is that it's writing resources that the kubernetes api under server I could conceptually understand right. Maybe one is a C or D isn't installed, for one so doesn't mean they need to be the native API is, but they look and feel like kubernetes resources and I think most tools kind of all this eventually at the end, right like there's, since this is the native speak of the kubernetes api server.

C

Eventually, if you're gonna have to you're gonna create a resource, it's going to have to help put something that looks like a resource the next one. Not all tools do- and this is what the input looks like, and this is that the primary input must also make you pronounce resources. This is then put in the output are symmetric. It may read open API in addition to kubernetes resources. This follows the conventions of what the like Princeton's. The tools.

C

Do coop control reads the open api from the api server and the resources also follows what the how the controllers tend to work where the controllers are also reading resources from the api server, and so you know this is kind of the the guide check here is: is it to control get a bowl like the inputs? Are there something you can get from to control? Get, for instance, do they look and parse like that or do?

C

Are they something that you get from github kubernetes examples and do those are all the are the cumin examples like functional inputs to kept the sources it can read from files standard in API endpoints get so again where they happen. To read the resources from is less important.

C

That's more important as this reading resources and then the third point I think is especially noteworthy because it's it follows what the controllers do, but a shift from many wet traditional, the way traditional tools of operated right, which is that it needs to be able to read what it is written in, perform like level driven updates sort of like a reconciliation rather than just regenerating everything from scratch.

C

So this is the sort of the micro services architecture right where you have different modular components, reading and writing data from some back-end or from the api server or that sort of thing and interacting by reading what each other written and reading what they've written in the past, rather than sort of a model where it's having a central piece that generates a bunch of stuff from scratch about on its own.

C

And so what do we get from shifting the model left right, so they can run in CI, CD workflows. You can script configuration changes that can then be reviewed through PR checks and rolled back through it.

C

You can do different development workflows around like ID integration and validation. So IDs already have some validation based on the open API, but you can imagine your what we'd normally be in a validating. Webhook wouldn't be great if your IDs could just capture that same validation and apply it right at night, II as you're developing, rather than having to actually try and push something to production to catch that validating.

C

What's not gonna work and one advantage of the ship left approach is that the scope of what it can validate is is greater because it can validate holistically all the resources that are gonna be applied. The bundle where, as validating Web books, only work on a single resource at a time so trying to catch something, a validation like this deployment, plus the service buses config map. Much look like this collectively is gonna be harder to do with the validating web hook than it would be in ship left and then.

C

Finally, on operations, the ship left provides a nice solution for doing cross-cutting transformations, injecting and it containers these sorts of things. This can also be done with mutating web hooks.

C

A lot of the working cap can be done on the server side right, that is by design that's the architecture, but there are reasons that maybe you want to do it on client side, sometimes or at least ship left and see ICD or before you're, trying to push something production right and you can visualize the changes, for instance, in a DIF with ship left, and you can have them reviewed in rolled back pretty with pretty clear workflows, whereas, if you're doing a mutating web hook that operates on a pod to inject an it container, and that creates a problem.

C

How do you, how do you roll that back, right, it's more difficult and so really kept is doing providing low level configuration machinery or that's its focus initially right, and so this is packaging, get plus animal.

C

How do we do that low level piece of not copying and pasting, be kubernetes example right and you can get clone that thing, but there's some challenges like it is not designed ordering around resources. If you want to do an update, for instance, it's going to do file based updates, as opposed to resource based updates and these sorts of things. Another level piece of does a static modification of configuration. So how can you?

C

How can we define things like set image or scale, or these sorts of things which read and write configuration files, but basically statically, like I, know how to set this value because it lives at this particular location or that sort of thing and.

D

Then finally, dynamic.

C

Modification of configuration right, so these are things where it's not simply a matter of go, find this field and set its value to this, we'll go find these different fields and set their values based on these inputs. It becomes more like this is more the controller based approach where it's like: okay, look at the holistic system and then what the desired state is based on what these are specifying and then figure out how to make it.

C

Look like, like you wanted to write like the program thinks it should and then finally, there's an actuation piece which is reading and writing from the API server. These are the extensions or they're like apply and status, and these sorts of things which we're pretty familiar with all right. So now, I'll give a quick demo.

C

C

Let me give a quick time of kept I'm in this kind of empty directory. Here, I'm gonna use my magic to kind of run. The commands for me so I can talk as it has a tightness the commands for me. So the first thing that's gonna do is fetch a package from git, and so this is saying go find in the subdirectory. From from a little git repo I wrote for this example to pull down the configuration right and so at this particular version on in typically in a workflow.

C

We want to go ahead and add this to get right away and that will rake the diffs easier. I can use git diff to come in. Show you the changes as we modify this thing and so kept has some commands for just viewing package. Configuration here happens to be one of them that allows you to kind of quickly at a glance, see what, if I fetch straight.

C

But if we look at what is actually fetched over here, you can see that it's basically a gamma configuration file.

C

You, and so following that it has to apply to a cluster thing like we can just to control, apply this thing. Capsizes own apply utilities which allow for some some nice functionality, but right now, I'm going to use cout control just to demonstrate the philosophy again, and you can see that there is this @ çd thing running.

C

And so now maybe you want to like modify this thing locally, and so this is not something you necessarily have to do, but.

C

When one may want to modify their the package that they found and add some stuff right in, and this is demonstrating that it has to be able to read what it is written and do level based transformations rather than like regenerating from scratch.

C

So as we modify this thing, it should be able to keep those things, because it's reading and updating so we're gonna do here is we're gonna, go ahead and update this package from upstream to a new version, and this should go ahead and keep the annotation I added while pulling you in upstream updates, because it's doing a merge right, and so you can see- and here that it's just changed this to a five from a four. You can see that change here as well. So that's what it's changing the update!

C

You can also see that it's kept that annotation bar again reading what it is written in, doing updates to it.

C

Going head in behine and the video files is, it's not always the best way of not always the way people want to edit stuff people do or there's different reasons you may want to just have put commands to go and change these things. It allows tools to do kind of the heavy lifting for humans so that it's less likely humans are going to make mistakes, and you can see what I've done here is just add. It run the set command. So these.

D

C

Notion of setters, these are the static transformations. I was talking about where it's possible to define sort of static ways that the configuration may be manipulated.

C

In this case we set the replicas, and then maybe we added some metadata about why we set it that way and who said it, and so now you can see when you list these setters, that replicas have been set from 1 to 3 and you can see I set it and kind of why I set it that way and that changed one field, there's also a name field which has which will change ten different fields of view set that one.

C

And so you can see here, what that's done is it's gone and it's just updated the yellow files so again going back to that philosophy, but read zml of right. Tamil reads what is written previously, and so this is loosely coupled with the packaging piece. Like you, the packaging piece reads and write animals. The setters piece reads and writes animals. They don't really need to know about each other for how each other operate, allowing a better interoperability, and so this case has just gone ahead and updated. That and I'll talk a little bit about this.

C

This open API a piece actually is how the setters are performed. The inputs are open, the API, the outputs are open, API and so we're using an open API to describe metadata about the configuration just like kind of kubernetes uses open API from the API server to.

B

Describe metadata about.

C

The configuration so we're gonna go ahead and apply this thing. There changes and we're going to see that actually, it's not working. There's these errors right and that's because uh we have designed this is that while it changed the replicas just changing the replicas, isn't enough right in this case, doing a pure static transformation which would work great for certain things like image and replicas, maybe for deployment and other things in this particular case for the state will set its saying.

C

You can't just change the replicas, because the other replicas now are not configured correctly with the right set of Enosh. The initial cluster configuration we're going to fix this by adding dynamic configuration, so I'm going to go ahead and pull in an upstream update.

C

D

Update is going to.

C

Add a couple things: it's going to add this function, declaration in the different function, runtimes and this one happens to be written in start, which is kind of a experimental runtime that were that hasn't really been supported yet, but but I think demonstrates very well the the architecture that we're trying to show here, which is because.

D

C

You to quickly perform a dynamic sort of scriptable transformation without executing arbitrary code on your machine. There's other runtimes that allow this too and containers are actually the primary one in time we're supporting, but this one happened to be a little bit easier for the purposes of demonstration, and so it's updated. This configuration to say hey. This thing now has this function, which is a dynamic, dynamic piece of code which should perform transformations, and it's done through kind of this Starlog script.

C

Here again, this is a Starlog piece is, is experimental and really great for demonstration, but the containers are the more mature of the two, and so this time we're going to set the replicas to two we're going to see something different. This time.

C

The replicas have been changed, but also this value of initial cluster has also been changed. You see, and so that initial cluster is instead of just being statically defined as the first pod. It now includes all of the different pods that we expand, based on the replicas.

C

Delete that pod, for some reason, is having trouble pretty starting, but you can see that it's running it successfully here, but now that we changed that and so I'll show you quickly with the reconcile function, looks like it's, the smallest snippet of code.

C

What it does is it takes in a list of items and then this this schema is well defined or the input output. It is well defined for what functions look like in this case, and it takes in the items this way and then so it pulls out.

C

Those different items in the end goes and finds the matching staple said, based on the name of kind of what was annotated with the function and these things everything goes calls out: the replicas and the service name and the port, and then it figures out what the list of pods are that are going to be part of the initial cluster based on that, and then it goes in it sets here.

D

C

D

Environment variable.

C

That list of pods is the initial state right, and so it's gone, it's gone and modify that particular resource in the items and then kept itself will then write that back out, and so this is automatically triggered anytime set. His is run right, so, by virtue of setting this to you, it runs this function in that functions. Job is just to make sure that all the configuration looks correct, based on its current set of values.

C

And then I can show you to the open API for the setters. This is kind of how the senator to find miss Donnelly to open API definitions which describe like here's, the name of the sorry, here's a name of a setter called replicas. Here's who it was set by and here's its current value and so going back to that CD AM will file. This is just an open, API reference, and so these resolve using standard, open, API libraries there's nothing specific about this reference or resolution.

C

The only things that are special are that kept looks for open API as comments, so as it parses the mo files, it looks. Is there an open API definition? That's comes as a comment on this particular field and then it also parses additional open, API definitions from the kept file, and so using those two things together, you can augment the open API provided from kubernetes with your own extensions, and then you can customize.

C

The individual object is open, API to be something more than just like the generic open API for that type that you could say for this particular object, even that these are all these particular fields, look which opens the door for and the future doing things such as saying this particular. Maybe image should have this regular expression and putting restrictions on certain fields for particular objects, while still allowing end-users to do modifications to them in customizations.

C

Okay, that's my guess is Morton up next gentle. Should we do questions on that first or should we do Morton's demo.

A

So I think Morton has some demos do. Do you want someone to take questions first or do you learn.

C

What are you in here I.

E

Can just run through it? It's uh it's just a smaller part of cap time what Phil went through, so it shouldn't take too long great.

B

No more questions yeah.

E

See my screen here.

E

You see my screen.

C

How what do you mean like it's? The text is just small. That's all it's kinda hard to read.

E

I think I'll have to do it like this, because I'm gonna demo, something which might might if I change the font size. It might not work too well. Yet, since it's still only so-so, it's gonna show it's the kept live functionality and it is for applying cat packages to a cluster and but kept like it. Just really.

E

It's just conveys resources so just like Phil demoed and kept we used in control, I'll, just demo kept live with regular resources, so I just have a set of five different resources here and I want to apply those to cluster and kept live adds a couple of things on top of what you get from queue control, so it has pruning which is slightly different than how pruning is turning cube control.

E

It adds ordering resources, so it make sure that if there's namespaces apply them before things going into namespaces and it also handles resource status, so you can apply something and wait for it to reconcile.

E

So first thing we need to do is because we're pruning works, we create inventory objects as part of apply, and to do that, you need to know where we want to put those inventory objects and also generate an ID. So first thing you need to do is just do kept live in. It now create an inventory template, which is just a config map. I can show it here.

E

Just contains the important thing is namespace and the annotation at the bottom.

E

Now can do then is I can do a preview. What will happen if I apply? This second is to kept live preview.

E

I'm gonna show that little create pod disruption, budget, a cron job, and you can see the inventory object on the top and then I can apply. This then I wanted to wait for reconcile.

E

This means that it will wait for resources to reconcile before it it blocks until all the resource, a miracle cell and then it'll print status as it goes.

E

You can see it crated them first and now it prints changes into the deployment. Has all the eight replicas already I think we're waiting for the last stateful set.

E

So and there everything is roll out so now everything is not only applied, but it also. We know that it reconciled can also get another view of this, and this is why this is new. It doesn't handle softwraps properly. So that's why I I needed a larger screen.

E

So this shows all the resources it shows the traits that are generated and the status conditions and some information about all of them we can do is now is we want to update something? Let's say we want to change the state full set here say we want to increase the number replicas to 4.

E

Then we want to update the version I.

E

Can do now, I can just run that apply again.

E

You can see here how the state whole set now will start creating pods and then we'll see as it cycles through first ad, we'll see it live number three and then he'll go back and add or delete and recreate the only ones.

D

E

You you can see here that the way staples are works. It will create a new ones and it'll go through each of the previous ones and terminate them and then recreate in, depending on the ordinal of the pod.

E

So this view is partly inspired by the cue control tree command. Okay, so this this was a timeout. So now this didn't I forgot to set the timer for longer than 60 seconds, but they still still. The record stall was still happening in background and we can do now is we can just delete a staple set and then we can do a preview again.

E

And then we can see here that the window run apply next time. We expect state or set to be pruned the service, and also the existing inventory object, as it will create a new one and if you do apply again, you'll see that the state flip set is no longer there, and this will be deleted and background we're working on functionality to wait while tuning happens and, finally, what is possible. You can also do a fly preview.

E

If you do destroy, then it will destroyed. Mine is a separate command which will take all the resources and just in this package and the leader from the cluster, so I can just do a preview and obviously, if I, to destroy he'll just delete all of it.

E

Yeah I think that's it.

E

A

Any questions yeah so.

B

With the functions that you said that are able to update it, you use open API specification, specify a call all into a particular runtime. If I understand correctly.

C

Sorry, can you say that one more time the audio went out halfway through? What's your said, I am in so ok with.

B

The functions that you use to do the updates right, like the gold go callbacks ahead called onset, for instance, it was are specified via open, API grant.

C

There's a couple of ways as know the functions are specified via open API with.

B

The reference to call the function is an open, API right or no, no.

C

The so there's two ways of there's a couple ways of running the functions, and it's there's, there's layers and the functions so so at the layer you're talking about is effectively. How do you, how are they orchestrated right like what causes them to be invoked and what I there's a number of ways of different of specifying them and to be orchestrated?

C

The one I demonstrated is through an annotation on a resource that says that that actually describes how to invoke the function like how to set up the B, runtime right and so for containers that would be like does. Should you enable the network or like by default?

C

The network would be disabled so that it so that it's relatively safe to ride, and it won't send your who face to some malicious endpoint or something like that, but you can say: oh, we need network to be enabled and then that's possible to do or those sorts of things those can be set on any those are set on any resource.

C

Then it will cause the function to be read with the resource that contains the annotation to be provided in a special feel in the influent called function. Config, and so you can build it. You can design a system to look kind of like a controller right where you could create, for instance, a new client side, C or D.

C

If you win is the way a lot of people have described it right, where you have some abstraction type and then you could put the annotation on there and then that would cause cap to go invoke the function, providing that sort of abstraction type as input just like a controller, C or D model would work and then the rest of the resources that are scoped to it.

C

Those those being in the same directory or sub directories would be provided as items and then the function would go and look at that function can fake and then look at the resources and then make sure that the whole system looked correct.

C

What I demonstrated is a little bit different model than that, so, instead of doing an abstraction right, where it's like a new type, I demonstrated a function on like the actual resource itself and then the functions job is to validate that that individual resource dude, but it's also possible to run functions like imperative ly on the command line. You can give it the like a dash dash image, you can say kept so I've.

C

Also I also just demonstrated an implicit run where the function is run after setting just just a normalize everything there's more of an explicit run where you can say kept function or kept up and run on a directory, and then it will reverse all the configurations and then run all the functions. You can also do an imperative mode where you can say kept append, run and then stay image, and then it will run the function from that image, explicitly passing in all the configuration into it.

C

B

I have to mutti, so let's say: I have one parameter that would cause effects across multiple resources. Would I have like one function that gets invoked from multiple resources in order to update those resources in sequence or what I have looks like you still get client-side CRD and then treat those resources as an artifact that's generated.

C

So you have one function that updates multiple resources like yeah, maybe what's a concrete example, maybe.

B

Let's say: I was dealing with like a typical stateless serving workload. I had like an appointment, I have service, I have pas disruption. Budget I've got a horizontal pot. Autoscaler, so put me so hungry and like I, want to make sure that the like I want to make sure naming is consistent across all the resources. For instance, so I update my hack name I want to update all of that consistently or I want to modify labeling across all of them and ensure that selectors conform to select the right, artifacts mm-hmm.

C

B

Yeah, how is that? How.

C

Would you apply that.

B

Right is that is that the question yeah? How are we the correct way to manage it? Would it be like having something that having specific functions for each individual resource that are triggered off of the same input or have one function, an outrage across resources, and so.

C

The off hand the starting with like the simplest approach that introduces the least amount of change to your existing system and then maybe only only shifting into divergent stuff when the benefit is clear. I start out with okay, what's closest to the system you running today. Well, if you have some ste ICD system already set up I'd like maybe create a github check or something together, part, whatever your CI CD system does as part of a PR and I run it potentially imperative Li.

C

That would be like the simplest starting point right and so you'd say kept up and run and then Imogen and give it the container on the directory and in that container, would be invoked with all the stuff and maybe give it some arguments into the function that configure it.

C

If you give it a command line arguments, it will automatically generate a function config based on those arguments, past that, and so then, from that starting point, then you get a cross-cutting view of okay validate this whole package, that's integrated into your CI CD system and then once you so then from there.

C

Maybe you now, you add two or three of these things right and then now maybe their configuration is starting to be more complicated, where it's not just like make sure all this stuff looks good and here's one or two key values, but it has a spec with nested fields and list items right.

C

That's where you start to look like it starts to look more like an abstraction type and instead of people just running on the CI CD system, where it's embedded into that script, you want it to be run as part of your make process whenever someone runs runs locally and then that's the point where maybe you want to?

C

Let's start looking into do we want to declarative approach or if you want scoping of different functions to different sub directories like you have a package of 20 resources with different subdirectories, and you want this function run against one subdirectory and that function really goes different sub directory, and then you want to run one command that does all of them like these are I, send more sophisticated techniques and needs aware, then it makes sense to do them declaratively.

C

But if you just want to say here's a collection of resources, please air out, instead of allowing a PR to be merged, if they don't look good, just running them, imperative, Lee I, don't think, there's anything wrong with. Certainly, okay.

C

But but that's probably, it depends on probably the rest of your she IDs process to write like so going with the principle of its maybe best if it kind of fits into what processes you already have like. If you're, if you have a CI CD system, that runs a bunch of checks like adding that check imperative Li there if everyone's gonna understand it, that makes sense right. If you don't have that system in place, then maybe you don't want to set it up and you just do it as well.

C

What I do, in that case right is I, create a new configuration file for your abstract type, maybe and then at kind of the route or above the route. Add the annotation man tation would describe the container or starlight script or whatever runtime you happen to be using. Then you run kept box and run on that directory, and then it would invoke that thing if that fits more and you're better in your workload, and that makes sense too I.

B

Have another question, but I want to eat up the entire time if anyone else has questions.

C

No sounds like it sounds like go for it.

B

huh How do I do so if I want to use this? So let's say I have a package and then I have the package. This is beautiful. It across multiple environments, right, like my primary need for customization, is to specialize a package to deploy it in different AMD's. How does that work in this model?.

C

I'd say this: this model still so I guess going back to. If your saw, if you have a solution now you like, like let's say you're using customize today, to do it right and you have a different customization for each each environment right then it would work pretty much. It would be an it would be a value oo, add on top of what you're doing today right. So you would write your package. As you know, a collection of it would contain the different customization files right into your package. Maybe now includes all the environments.

C

I'll go through two scenarios to say you're the deployment piece. So then, maybe that's your that's your temp package, and so you run customize, build or control by care or whatever. It is against each of those environments.

C

What you might do now is like what do you do on top of that? Well, maybe right before you apply to an environment, you need to set the manifest of the image right that you just built, and so before.

C

Maybe you have, if you had a process where you set it in the the SHA of the image and that the image tag right now, you maybe create a set there for that right and so now, chromatically from you know your deployment pipeline, you can run set image and then run all plan after that would be one way of integrating these capabilities to maybe solve a particular need another one would be.

C

You could, hopefully the actually remember if the there's a way to the function so that you can like do customized build and then pipe it into a function, for instance, so maybe right before doing control apply, you can customize build the environment, pipe it into a function, make sure that that exit successfully and then and then apply after that.

C

What are the problems you're trying to solve? I.

B

Was I was trying to understand how it fit him with overlay customization for multiple, if I wanted to use overlay based customization of the multiple environments- and it sounds like primarily, it's a packaging system there's a value-add on top of that- that's compatible with overlaying customization, but doesn't mandate that particular approach.

C

Absolutely so maybe I'll say to two things of how maybe this fits in with customized. The first is even if you're, using customize to use bases and such there's still a bootstrapping problem of well, how do I get that initial customization guy yamo right? That says which bases are I'm using right or what the different you know.

C

Common annotations and common labels are right, and so you probably end up copying and pasting that from some location right to get your initial state and then now that's kind of orphaned from wherever it was copy and pasted from, and so, if you want to update the version of customize, for instance- and that means some new feature was enabled- and you want to add that to the customization file.

C

In the example like, how do you get the latest version of the example right, so the customization gammel itself a package and then kept getting that and then just doing your normal workflow does solves a couple of those problems for you. The second is it's a blueprint customization, so there's the environment customization, which you I think or just talking about, and then there's.

C

Maybe a blueprint customization, which is okay, I, wanted just like three Java backends right and you can use to customize to do that using different bases and creating different graphs, and some people really enjoy that model and they like it and and they totally get it in which case keep going that route right or I'm. It's not wrong to do it that way, but the one trade-off there, as you end up with kind of a large graph right and I've, seen some very deep customized graphs and then folks, you're kind of limited.

C

What can be done within the customized framework? A customized framework is very flexible. So it's not like there's a lot of limitations there, but you are working with in this graph now, and so maybe, if you're like saying, for whatever reason.

D

C

Don't like that approach- and you just want like hey I- want three copies of the back end and then I want to manually customize those things instead of using patches and I, just want to pull them down from an example, and so what you would otherwise do is copy and paste the Java back in yeah. Well, three different times, and now those are your customized bases right now would be kind of your blueprint approach.

C

Well kept now allows you to pull down those different Java backends into different local packages, and so, instead of having a shared base between a shared Java base that you try and create this customized diamond-shaped graph with you, you no longer. You have a different sort of graph where each of those is a completely independent node and there's no shared base between them through the customized graph.

C

But when you want to pull down updates to them and use kept to kind of do that resource based merged model of pulling pulling updates from wherever they originally came from in I, guess, they'd, say the way I've described customized and kept to folks it cuz. Some folks have asked like well, should I use customized or kept for this.

C

There seems to be overlap and and what they're capable of doing or the problems are capable of solving and the metaphor, I upheld in my mind, is kind of like, but in a programming language you have both while loops and functions. Right and technically you can write a program without functions and just use like ifs and wild loops.

C

Right and technically, you could probably you can write like programs without while loops, if you get like some crazy recursion right with functions right and so, while both functions and while loops have overlap and when you, when you've, have a particular problem, you may say: should I solve this recursively with functions or should I solve this with a while loop in a stack right on any given problem and then the answer is, you know it depends on the problem and what your programming style is and what happens to make sense there.

C

Yes, there's overlap between them. Yes, it makes sense to have both of them, because they provide a slightly different ways of solving kind of the same problems that be can be used together. Super.

B

Stateless server, a good example I. Imagine that you might have like a blueprint called service or micro service, which contains all the resources, and you would generally have to start with it. You can use that print to generate yeah mole, that instantiates specific service give it application, name all that stuff, and then you could use customize for last mile customization as you deploy that gamble across different environments like that was yummy, but.

C

Pretty much yeah yeah, that's that's the model, we're considering and then you can even throw in functions like there's. We're gonna have to get more opinionated here. I think what we're gonna have to do is build more like very opinionated workflows and maybe there's probably gonna be three or four different patterns of like. If this is how your musician does something, here's how you do it, but you can even use functions on top of that right. So you could use a function with an abstraction that generates the config and that's your package.

C

Your package could just be the raw mo for the stateful or stateless service, and then that's your packaging, pull that down and modified in place rather than have an abstraction and either one of those cases. You can use the eval to the function or the raw base as customized basis and then do do variant customization from there.

C

That is what I say. What I just said makes sense.

C

B

Any other questions I.

A

Have a question about how kept live? Fern is different from the one in Q control.

E

So the prune in kept is based on creating an inventory. Every time you apply and diff inventories against the next sort of several resources. The one in control is based on labels, I think. So it's a slightly different approach.

E

A

A

Any other questions.

A

Think we have four minutes left, we don't have other questions or topic. We can end the meeting now.

A

Thanks everyone for joining us awesome.

B

Demo, film and morning Diana. Thank you. Thank.

C

You thanks for having us.