Grafana Tanka, 6 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tanka Community Call 2020-10-06

Description

We lost the first 15 minutes or so from this call which is why it's starting in the middle. Sorry for that.

A

Like the service monitor mechanism and all of that, but for monitoring our own infra, which includes cortex loki on the clusters, we want to deploy all the entire stack additionally via tonka, instead of via the coreos, permeate his operator. So many of the charts that we're installing today have either service monitors or other endpoints that need to be scraped. So it seems like I need to learn a lot more here, but but this is a bit of a game changer for us in a good way.

B

A

B

Fantastic to hear uh we can say he was actually chart manager that uh assert manager. Rather, that was the one that led us into this is such a complex chart.

B

uh We have forked it, and then we dug ourselves in holes, and- and this is where how we got here.

A

uh That's why we did customized for the same reasons for running cert manager, with let's encrypt for all of our stuff, along with external dns to cloudflare, so those two form the basis combined with nginx of all of our ingress management, insert management and yeah all right.

C

I've I've just been working on deploying a lot of our internal clusters with flux um and the way we're doing it is not with the the transform functionality that's going away. We actually we check in all of our tonka environments, to to our main repository. We have a ci job that runs essentially tk export in parallel and spits them out into another repository.

C

That is just for the raw kubernetes yamls, and that is what fluxwatch is and so really that repository could be generated by tonka or helmer or anything else. He wanted to it's kind of just manually doing that that transformation phase, so flux doesn't.

A

Have to that's what we're doing with cortex today, but we didn't have time or energy to learn: jstaunite and tonko when we had they put flux in production last quarter, so we emitted that once checked it in and we have a whole bunch of customize. On top of that to add in annotations and things like that, and in the last two or three days I've been backfilling that and and converting that all to tonka. So I don't have a nasty three-way merge every time.

C

A

Are still kind of fighting through.

C

A lot of the logistics of okay, we, our main workflow, is you export it and flux takes care of it. But then how do you also have like a tk diff workflow for development, and can you do tk apply on an emergency with without flux overriding it so still kind of working through a lot of this, probably a few blog posts to come out of this once we figure out how it all works.

A

We're doing the same, we're working on a github action and then same same challenges. I'd love to talk when, when there's time outside this meeting, if you're, if you're up for it but yeah, that's active work right now. Last night,.

D

Sounds very cool all right, I suggest we could go on with the next topic. So malcolm do you want to say something on jason, the tips.

B

Yeah, okay, let me share my screen.

B

So some of you will have seen this before internally, but this is something I want to make a blog post about.

B

um Fortunately, for the people here, I've realized that the complicated bit uh was unnecessary, so I removed it. um So basically uh what we've got this is um a an attempt to communicate some of the lessons we've learned from using jsonnet internally as a language.

B

um You know, I could say: we've inherited some ways of doing it that have not served as well as they could, and these are ways we've worked out that make that use the language more effectively.

B

So first thing to say: uh this is not not intended as an introduction to jsonnet. It's assumes you know jase on it and have uh have some experience of actually you felt some of the pain already.

B

uh So what's the problem, uh so it's a very flexible language. uh You know perhaps too too much so so it allows uh to create unstructured code. um So how do we? What do we do about this? uh Well then, it's matt saying he's three days in. Hopefully this will really help um so so the way we solve that is by adopting some conventions.

B

So you know, for example, in golang we could just write our entire program in one large main method or one large main.go file, but we don't so in any language. We can adopt bad practices, but we don't because no one ever does we copy what everyone else does so in a way this is presenting uh some conventions, so we're not going to cover the journey here. I forgot to remove that word. uh Let's get let's get going and have a look at it.

B

So, first of all syntax, so let's just have a look at a few things to do with uh syntax. That will be useful to know. Here's a stupid one uh that tom told me a while ago, colon we might think colon means public it doesn't. It means inherit. The default is public. So when you use colon you're you're picking the default value of public, so colon colon means hidden.

B

So if we say this, which is hidden plus this, which we assume means public, we get this which might be somewhat different to what we think we would get and that's one. We could easily find ourselves bashing our head against that for for hours. It's a very simple solution, use the triple colon and suddenly you're saying take the hidden thing. Make it visible.

B

Kablam you've got your nicely merged chase on it. Okay, that's the first one, uh second one! uh If you're debugging! Well, you can use mine the the tk eval minus e, but sometimes it's nice to do. You know you add, as you're writing golang, you might add format.println uh etc into your code. uh But how do you do that in jsonnet?

B

uh Well, one way you can do it is emitting trace messages, so all this will do is print some message and then it returns the value so effectively. This whole thing here is equivalent functionally to that chunk there. So you could include in the some message.

B

uh You know a json manifested version of your data as well yeah, uh but it can also be used for deprecation messages, as seen in this example. Here um commas and pluses, I'm not sure whether plus is as a two s's or one, but I think you'll get the point. So if we look at this code, uh what we're doing is we're taking my lib.

B

Dot new creates me a new version of my lib. Then we add on to it. We call them the my lib, with with a function called my with something uh app which gives us a patch to the original, and then we do another one that adds something else on. So this is all perfectly reasonable, except when we comment out one of them.

B

We get invalid code because there's an extraneous plus on the end, and you can end up spending lots of time faffing with those extraneous characters, whereas if we do it, this way, stick the pluses on the beginning. Suddenly you can comment out a line.

B

And it doesn't it's really not a problem now. What I do accept is there's likely religious wars over whether that comma should be dangling or on the end, yeah. I think we could have many many arguments about that, and I'm quite happy with that.

B

But the idea, at least of having the pluses at the beginning makes life so much easier, um extensible library so creating how to create extensible libraries.

B

So uh a few things so I'll say these briefly, but this course being recorded so uh and- and I will be making a- I believe I'm gonna aim to make a blog post of these points. uh At some point, so, first of all document, you there's a tool that tom's created called docs on it, which you can use for documenting your libraries uh provide a clear config.

B

So how do you configure your library and how do you make it easy to extend so so I think what I should say for this audience particularly, is what is a library uh um so exact, for example, a library could simply be. You know I want to install mysql, so I'm going to have a library which could either be in my slash, lib directory or vended in from github in my slash vendor directory and it's a an encapsulation of some functionality.

B

It wraps some functionality that I want to read to consume in multiple locations. So if it's spended particularly, we might want to make it public, we might want to document it and have public documentation about it, hence doc sonnet.

B

We also want to have it having clear inputs and this being jsonic. We want it also to be easy to extend. So, let's have a look at those three. So here is the current version of docs on it, which takes a little bit of.

B

Picking apart, so if you see we've got an element here in quotes with a key in quotes, beginning with a hash and it calls d, which is the docs on it library. It says the thing that comes next is an object and here's a description of it. So the next thing is an object. It's called my obj. Oh here is my obj uh and the same thing here. The thing that comes after me isn't my funk.

B

Oh look, there's my funk and we say: look it's a function uh and here's a description, and here are the arguments. Yeah and the arguments is just uh so. The first argument is a who and we're telling it's a string so suddenly json it becomes typed yeah, uh and you can see an example here of archaic, alpha library that uses this extensively.

B

uh Here's a simpler version using comments. I don't know thomas this- is this app public yet or is this still in development.

D

It is public, but it's not merged yet so it's a branch on the project, so it needs to be merged.

B

Into docs on it to do this, okay, but yeah, I think you'll, look at your look at that and think actually that's much clearer and the the structure here is static, but but certainly can so you can certainly start using it whether we've finished the code, yet the box on it code yet or not so, moving on to library configuration.

B

So how do we pass in so a present? You know this. This is a common practice, is to use a global route dollar, add everything together and then reach it from a library. You just reference dollar and you go, get go, get a value from the root and that's just a that's just a mess. It creates really complex code. So what we do is we make sure we encapsulate the library with clear means of invocation and passing everything uh to the libraries rather than referencing the dollar, but bear in mind. Dollar means.

B

The root of this particular tree is not the root of the whole tree, so we may still use dollar and that's referring to the root of the library, especially the library, is within a local, so dollar doesn't that dollar isn't always evil, although it can be sometimes so. Let's look at our library here, we've got uh so I should just say that dollar here I can, if you see here, dollar, is referring to the root of the current tree.

B

So uh here we have dollar dot, underscore config, it's referring to dollar, dot, underscore config and then dot replicas goes and gets this value. So here we're configuring a grafana. You know this is very much pseudo code, but and it has one value, grafana replicas and that value comes from a hidden config.

B

Then we have a function called image which is hidden uh that has that returns a single element with a key of element, every image and a value of whatever we passed in. So that's how we. This is how it's typically done. uh We create a local, but we add local to the root. There's an implicit plus here. So we've got this block plus the grafana is all smudged into the the root, and this is what we're suggesting we don't do so. uh The if you see here we've got plus we've got grafana plus we've got.

B

If you see we say whatever comes out of this. This will return a grafana element and to that grafana element we're going to add an image, the output of this function.

B

So we end up with grafana. That's what was in there before and with image. It adds in the image, so you've got this nicely merged output so plus has merged. What was there before here with what, with this here.

B

Yep, so we, if you have questions about this, please note them down happy to answer them at the end. So this is a another way of doing it. So here we have local grafana.

B

Instead of putting it adding it to the root, we create an element called grafana and we call we import the function in here and then the implicit plus means this plus this. So we add another config element. This plus means don't overwrite. The element merge this config into whatever was there before. So we can just override specific config elements. So here I'm setting replicas to two.

B

So instead of having replicas one, we should have uh replicas two instead and then we add on to that grafana dot with image, and we get this you'll notice in the tree. There's two is grafana grafana, but that doesn't matter to tanker because tanker strips out all of this, it only looks for the kubernetes objects anyway yep, so the json might be slightly different, but that's really of no consequence.

B

So, moving on to patching uh so now what we've done is we've consumed. We've executed, invoked library, we've handed it configuration uh and we've invoked it now. We've got some some json back and we want to do something with it.

B

So as for example, tom was saying earlier with the helm support the library could, instead of just creating js kubernetes resources, it could call helm which creates the kubernetes resources for us, but we then patch them. So how do we do that?

B

uh So patching json objects is easy. uh You just do plus uh patching lists is hard, so it requires you to iterate over the whole list, which is really annoying uh and messy. So there's three solutions. I'll I'll show you through them, rather than read them here.

B

So here is a library, um so this library simply lists products. uh Some products that we have, as you can see tanker, is currently more popular than grizzly, but I'm doing everything I can to to sort that um as you can see. So the problem we have is someone else has just clicked on on us to like the project.

B

uh So we need to patch this to show to represent this. Okay, so how do we do that? Well, at the moment, this is what we have to do. We say we load in our grafana labs library. We say our grafana element plus grafanalabs. It said grafana labs, whatever comes out of that library plus this patch and this patch says products, and then this array, and then it says for everything in super dot products. Super is the thing the previous block to me.

B

So go through everything in this here all the products in here and then, if the name is grizzly then increase the stars, otherwise just return to e. That does in fact work, but you try reading that code and debugging your way uh through it yeah. It gets really quite hairy, whereas if we so it requires iteration replaces all elements it's complex and fragile. So that's the problem with it have a look at this. What we've done is we've named.

B

We've created two hidden elements: tanker colon, colon lamp, grizzly, colon colon and we've created an a list from that. So I'll say more about this later but, like I call this late binding, you create the list that you create. The data that the result that you actually need late and you give yourself points that you can interact with it internally.

B

So now we just import graphvana labs, grafana labs, plus grizny, merge in this to whatever's there already it stays, but the grizzly thing is hidden, so we stay hidden.

B

uh And we put, we set our value of stars to 48.

B

Okay, I think yours, your degree, that that is a lot cleaner than that one yeah. So really it's a question of using some form of internal data structure, so this really works well, when the number of items small, uh when the number of items gets bigger, we have to do something else. So what we've got here is we've got a product map and this can have as many items in it as you want, and here, instead of just adding specific elements, what we do is we do another loop for t in.

B

We get the object fields that that is just the names of tanker grizzly iterates over them, and we just then go and get the values out of this array, and we can still do the same manipulation as we were doing before this time we go and edit product map edit grizzly.

B

Yeah, so this works well, when there's a large number of items you know in in here, you know you could add load in another library that has a whole load of patches and add that patch to the like, you know so much you could do here.

B

um What about uh when you when order matters hang on, did I skip something?

B

I think this is in the wrong order. It's got in the wrong order, so let's say what we want here is, I think, grizzlies more important than tanker. I would uh so, therefore, it needs to be first. How do we do that, because, because maps aren't ordered in jsonnet, so stolen from stanislav one of the maintainers of jsonnet, what you can do is you can maintain a product order, a separate list which restores the order. Then what we do is uh we iterate over the that list and then return the items and you get that.

B

So, and we can still uh using this, yes, the are in got in the wrong order. um We can now patch it in pretty much the same way as we did before.

B

Just to say, I did say that this was not an introductory chase on it presentation it's pretty full-on, but it's really useful things if you're going to be spending time making lots of libraries. So this works. When the orders of items matters, uh we might, what might most likely want to create a little library so that, when you add an element to it, it actually adds the um the library will add it both to the map and to the order.

B

So rendering late or late binding, uh so I've said this here. So this is the thing we really want, but what we're doing is giving ourselves these, but this thing is really hard to patch, because it's a list- and it's just messy- and maybe it's a kubernetes object- and it's like 15 layers, deep and so you're going to have to do this. Stop this stuff or yet it's horrible tree. What we can do is we can expose.

B

For example, there's a common use of this is in containers in kubernetes uh is put the container so mysql underscore container colon colon as a hidden element and then in your deployment, dot, new square bracket and you list dollar dot, mysql container and suddenly you've now got an entry point where you can say: oh add this environment variable to this container, etc, etc. Suddenly, you're, rather than embedding it into the deployment.new, it becomes uh patchable.

B

uh You know here's a rough example of you know an example of how that could work. You know we're iterating over the same thing. This is not valid code, but you get the idea. We we can iterate over all of these and and create our config.

B

Maps so uh moving on to the next thing, mix-ins for extensions. uh So if you let's say, for example, we have a I've seen this number of times when people say they want to extend a library, so we have prometheus case on it and they say something like if conf config is h a then do this. Otherwise do that in jsonnet. What you end up very quickly, is you end up rewriting helm charts, uh which is just this smudge of?

B

If this then otherwise do this, but not that and oh, but if you do this and that, then you know if it becomes very confusing very quickly a much better way to do it is to create a mix in so prometheus creates a default prometheus and chromatin case on it mixing in set up, but here we add on this mixing so mixing mixes in to something.

B

So this is not a valid resource in itself. All it knows is how to convert this into something that does prometheus h a so. This will install a single prometheus. This will install a prometheus h, a pair, and this way you can have mixing after mixing after mixing enough to mix in, and they can at least in theory. If you construct your initial library right, they can all just merge into one happy family.

B

uh So what this also does is this keeps the knowledge about how to adapt your library in the library. So what it's common to do as well is you you have this library, which makes you a thing and then over in your main.json. It you add this huge jsonic patch, which is not actually consumable by anyone. It's like you can't make sense of it.

B

So take that put it as close as you can to your library, and then you call that as a mix in so you just say, uh mysql plus mysql, with some super feature, enabled and then then your I'll say more about that in a minute avoid this pattern, which you may have seen on: kubernetes libraries, dot new dot, with new dot with something else etc, because not because it's inherently bad, but it just breaks the jsonnet interpreter and makes your code much slower.

B

So use this model, even though it's slightly more verbose. This plays to the best of jsonnet.

B

So a couple of last points, some of which I've made already try and make your main.js on it as clean as possible. Put your push your logic as much as you can close to your libraries uh you're in invite consider your environment to be your specification. Your environment says this is what this particular environment should be doing, but you don't really want code in that.

B

You want to push that some as as close to the libraries as you can, potentially with with mixins. So you know with dev mix in says: okay right, remove, authentication or without without authentication or room, you know, etc. Mix-Ins.

B

So, let's keep going so first of all push logic close to the library. So here we have grafana plus grafana, with some feature really clear what you're doing yeah I have a grafana and I have a graffana. I've enabled a specific feature and it's easy to see how you can disable that feature now, and you don't have this uh situation where you've got a huge blob and you don't really know what it's doing or we could, let's say we're doing a lot of things in a main.json.

B

We could actually in our main.json it separated load of have a separate load of files in our environment and import them, and we then know that all of the stuff in this file is related to grafana great.

B

um This is a rule of thumb for me. If, if you can avoid importing k.lipsonit into your main.json, I would say, you've done really really well, because the fact that we're deploying all of this to kubernetes should not be relevant to your main.jsonnet. It's a specification of what you're doing the fact that you're using kubernetes or kdot lips on it is really something that's more relevant to your actual library than it is to your main.

B

Abstract environment, something that tom named most libraries didn't create an application, such as say mysql, but what if you've got a website? uh And you want to have a staging website and a production website- that's hosted on wordpress. Actually, you can create a lib, but it's an abstract environment. So it's a website and what that does is imports the mysql lib and imports. The uh the wordpress lib and you're done.

B

And that would be an abstract environment that can be configured to do different things uh in different environments. So congratulations. You made it to the end uh any questions, but I suggest we stop there and open up not just questions about this particular presentation, but everything that we've been shown here.

B

D

Right question.

D

B

What the hell was that all about as a valid question.

D

That's good to hear matt.

A

Yeah, like literally last night, I was plumbing out um again we're deploying loki and we've already got cortex running and, for example, when we run promtal, we have a default service account that we don't use. So we make service accounts for everything, because the default pod security policy, which we named zzz.whatever because kubernetes is weird and alphabetically that one will be last, has no rights at all. So if somebody doesn't specify a service account, you get a pod security policy. That says deny everything right. So I was looking.

A

How do I patch in support for pod security policies and then make additional annotations or changes to the existing cluster role? Bindings cluster roles and service accounts?

A

I kind of teased it back through, but I've got things like that, as well as things like adding annotations, which I sorted, but my main.jsonnet is just like a gaggle of patches and I have to actually import the case on it stuff so that I can reach in and add an annotation to memcache or to console so link or d proxies, aren't injected or whatever, or we're also using annotations on service accounts.

A

This is the one that I still don't have solved, and I think one of your tips gave me an aha to add ekf aws, arn's or roles that are to be assumed by a service account via annotation, so that service account can, for example, write to dynamo or s3.

A

So some of the things you presented really framed it for me um again, I'm super new, so most of my questions are newbie newbie grade on aubrey.

D

Right cool, so tarask asked: is there a best practice to do conditional imports or mixing space on a var value? So I guess you mean an extra with that.

D

um I guess there are no best practices for extorts in general, because these are considered an anti-pattern by the jasmine project itself. So these were added several years ago and I have supposed to be challenging because when they are used for a project and set on may not be set, it might become challenging to understand where they are used and what effect they have on the outcome of the configuration.

D

So I would be interested to hear your use case, and maybe you could explain a bit what you're actually trying to achieve. Maybe there's a better way that does not involve extras at all.

B

And I guess I guess the scenario is simply do you want to you know if you want to say in one environment something switched on in some in another environment, something isn't it. Yes, I was in a.

E

Noisy office, I'm just going to get a spot yeah, so this precisely. um Instead, we.

E

Integration system has basically all the information about the environments and uh is basically the target cluster configuration and we're using, um and this is to stay compatible with some other patterns that we have we're using on these external vars to switch on and off on various features, doing testing and before things kind of go out right now, I have basically imports doing. If then, I was just wondering if there's a slightly better way to control um imports and make sense.

D

um If it's like more or less stable what you would turn over turn on for certain environments, it might make sense to actually have these as environments. So, for example, we have multiple layers of environments. We have a first layer which is application, and then we have a second layer subdirectories in the application environments which are regions.

D

So maybe you could have also an application, then on feature levels or something or if you really want to use late binding. You should probably check out top-level arguments because that's the designated replacement for external variables.

E

Okay, yeah, I got you that that's something I was actually looking into as well: okay, cool.

D

So, just on that again, the difference between extras and top-level arguments is that extras are accessible from everywhere in the code which makes it challenging because, for example, a downstream level you didn't even offer might react to an extra being set without you. Actually realizing and code might break if you set it or not, set it while top level. Arguments are only permitted in main.jsonet and happen at the very in a very defined way. So, instead of having an object at the top level of main.jsonet, you just need to add a function there.

D

And then you have function. Parameters on that function, which either has a default value or is required, and then the jsonnet interpreter prompts you to set that variable.

E

Yep makes sense. Thank you, cool.

D

Any more questions.

B

But it is worth noting that it is difficult to do imports dynamic imports. Jsonnet deliberately uh leads leads you away from dynamic imports.

E

Yeah, I'm so so noticed.

A

Yeah, I I have a similar question like so I'm doing all of my work now just on one one cluster but like I said I want to replicate this out across our fleet, and I don't want copy paste in in maine.json- is.

B

A

A notion of nesting with libraries that I should be aware of, like you know I could, I could take everything that I'm doing. That's like one of my clusters, which you know for for for you know, is really a bunch of scrape configs, grafana prometheus, just a whole bunch of junk plus some stuff that we're layering and that are patches on those things. So I'm already like importing libraries, but if I wrap all of my stuff, that's a cluster into a library and then reference that, from my dozen main json, that's one per experiment.

A

C

System, do I run.

A

Into some kind of, like nesting, recursion problem, where json is going to blow up, or is it just happy to just blah blah.

B

Blah that's the power of it is. Is it it's? What it does better than helm? Is it adds sort of semi, infinite, uh recursion and and abstraction?

B

So you can. You can build large data setups. So you can. You can have a a very small main that just says these are the key features that are switched on or off and then in your abstract environment. You have you describe that whole environment that whole name space and everything that goes into it, and by doing that, you ref you reference the libraries which give you your individual applications.

D

Right, abstract environments are probably the key thing here so to differentiate these from libraries. A library would be something very focused, for example, a library for mysql or a library for nginx. So these deploy a single piece of software and optionally set some default values, but not go much beyond that, while an abstract like an abstract environment, it's technically different from a library, but logically you define how an environment looks like without actually having all of the environment specific data in there. So, for example, say we wanted to deploy loki to four different regions.

D

We could have an abstract loki environment which holds all of the setup all of these four region environments should be comprised of, and then in the main.jsons of the respective real environments. We would only feed in very environment specific information such as secrets or ip configurations and stuff like these.

B

B

Right, you know, and another trick that tom's been doing as well is uh or showing me is, is you can create an environment, and then you can actually create another environment that consumes that environment. So it may be that you, for example, what taras was saying about wanting to switch something on and off. You could actually have a direct.

B

You have one directory which says here: is our environment but and within that directory is another one which contains a main.json which imports the what dot dot, slash, main.json and patches it in a specific way, and that so, rather than handing in values, you can actually call different directories for different outcomes.

A

So this is what we're doing with customize. Now um I mean the same metaphor, so we're running grafina enterprise, for example. Everything we do is staging and prod. So I have overlays for staging that just bump an image tag or something like that. So that's great well.

D

This is probably a cool thing for just getting done something quickly. I would generally say that if you want to build something sustainable, you should not have. One environment depend on the other, but instead do this to the thinking to properly abstract out the things that apply to both on a higher level and then both have to have them consume the higher level thing. So these are both first class citizens.

B

Yeah, that was the next.

D

B

You know the the the one important like one environment importing another is a useful tool for working around some edge cases, but not a best practice.

B

Use your english, abstract environments in lib for that.

A

Yeah yeah, I guess that's what I mean so with our customized configuration, we have a base configuration that most of the configuration is in and then we only have minor kind of patches to try on new features or enable feature flags or things like that. Yeah.

B

Yeah and you can make those and you can make those patches if they're, if they're just fixed patches, you can make them hidden on the main library and then add it in to the library and so so yeah or you can make it a function on the library so that it just can be then consumed.

B

Probably look at the when this cool uh recording's shown have a look another look at the slides I put up because it's all in there.

A

I had one last question: um I spent some time last night debugging the old hepty ovs code plugin for jsonnet and found the many ways it breaks. One of them is rendering webpages and the other is it blows up at stack, attempting to grok kubernetes alpha? Are there dev tools that I haven't found yet to either just make editing and or authoring json easier.

B

uh The the death of the vs code- one is a a real shame and if you or someone were to fix it, uh you would become extremely popular in this part of the.

A

World I forked it last night and started figuring it out and then I realized I had to go to bed.

D

Yeah editor, tooling, is a real problem and we all just use plainvim or emacs or whatever. We have at hand to modify jsonnet files. There are some plugins to at least add syntax, highlighting and indentation ruling, but that's mostly it there's no such thing as a debugger or yeah.

D

Well, the probably best thing to debugging json is writing code. That does not need debugging, even though that sounds like something magic like it's. It's easy. It becomes easier when you try to keep code simple, so not do a lot of computation avoid using if else try to use mix ins more.

C

A

C

A

Right now, I just do a tk export and a tk eval, and I dump them to files and the manifest. So whenever we change json it like the manifests also are in that same commit, so you can kind of see the station changed and it had these outputs and then flux picks up the raw stuff. But my debugger has just been running a script that does that and then uses git to just see what the changes are. It's a little. It works, but right sounds totally valid.

A

I'll see what I can do on the other thing, no promises on the plug-in definitely.

D

It'd be super cool to see somebody pick this up, because this is like not like we're frequently getting complaints and issues on the plug-in. We did not even write, but so apparently people really want to see it again.

D

A

My my partner, my girlfriend right here, is still at vmware, and so I know that the heptio folks inside vmware still like json a whole lot, but I think they're working on other things should we judge us.

B

There's still plenty chase on it in the world.

A

um I don't see a better choice. We looked at jsonnet and tonka before tonka. We looked at json at customizing home a year a year and a half ago, and I kind of cried a little bit when they mothballed jsonnet, because home kind of sucks um having used it at scale. It's it's brittle and awful. Just.

B

For the wreck, what do you mean case on it.

D

So no promises.

A

I'm sorry I misspoke yeah.

D

No problems with metis that this will ever be a thing, but there is something in development not from us, but from some probably very clever people called qlang which aims at revolutionizing config management using lessons learned at google. So maybe in one year's time this might be a cool thing, definitely worth keeping an eye on.

A

A

Awesome um is: is it just tonka or tonka dev, where the helm stuff is going to be talked about moving forward, I'm extremely excited. I know.

D

Yes, helen books are now on trunk of death, hell cool.

C

You can also drop in the grafana labs, slack channel and there's the tonka channel in there. If you want to chat.

D

Awesome, thanks sure, always cool to see people use it.

A

Yeah, I I think our biggest challenge still is unifying service monitors and the prometheus operator. I know it's not really a tonka thing, but I was kind of hoping that you know right now we're using that for everything. But, as I said, we want to cleave that and just let other teams use it, so they can self-declare.

A

But there's a lot of in for things where the helm, charts declare service monitors.

D

Oh when, when using the premium operator, you really should not use the helm chart but q prometheus, which is the json library of the same office,.

A

So exactly yeah yeah yeah, we were new, so so we were using that whole rendering the prometheus operator out just to get it working and then suddenly we had a dozen clusters, so we're refactoring.

A

But we have now, you know dozens of services, writing service monitors to say come scrape me, so you know that that actually is working pretty well, but for our own info, we've decided, as I said, to be able to unify so like we've got cortex, but keeping dashboards and scrape configs in sync is a bit of a pain, so we're hoping to just emit everything with jsonnet and then use it as a way to show our teams like hey.

A

If you're writing net new things, here's how you can bundle dashboards your metrics, your scrape configs, all together in a mix in, but I think we're pretty we're still a quarter away from knowing it well enough ourselves to teach it to our teams.

C

I would kind of throw in the caveat that we made the helm integration as more of a compatibility shim, so we could use helm charts rather than like. We think you should do helm workflows with this tool like I, I would definitely advise against having new teams. Writing helm, charts for use in jsonnet.

A

Yeah, I was talking about a couple things at once. I actually meant for new teams just doing just chase on it. I see this more as a migration path where I can go from.

A

You know a bunch of existing helm, running some core infrastuff and you know bring it into json and then over time because we're a tiny team at my company servicing the whole company. We can kind of piecemeal, page out, helm and replace it with jsonnet, but it sounds.

E

Like a framework.

D

I'd say the helm: integration of tonka is working far better than we originally expected, so it's definitely valid to use that as a migration path, even though to use the full power of tanka, you should definitely start writing as much as you can in json at some point. But for the time being, it's probably a very valid thing to use what you already got and onboard it onto tonka.

B

Yeah the way I framed it for myself is that we and we should we've running as we go over time. It's simply that if you're, if you're, installing something that you just need as a side tool use helm if you're installing your own application use, jsonnet.

D

Right as we are basically at the end and over time, I will stop the recording now.