Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Managing Applications in Production: Helm vs ytt & kapp - Dmitriy Kalinin & Shatarupa Nandi, Pivotal

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Managing Applications in Production: Helm vs. ytt and kapp - Dmitriy Kalinin & Shatarupa Nandi, Pivotal

Have you ever got a stuck Helm upgrade? Did you struggle counting number of spaces to use for indenting a template chunk? Have you switched away from generic deployment tools to writing your own operators? Do you dream about extracting common app configuration into a library for use by all your applications? This talk will dissect challenges you and I have faced while working with production Kubernetes environments, managing non-trivial application configuration, and deployment mechanics. It will dig deep into Helm v2, Helm v3, ytt, kbld, kapp, to understand design choices that ultimately result in great successes and failures in your production environments. Attendees will come away with a deeper knowledge of the inner workings of these tools and appreciation for the complexity hidden away in these tools.

https://sched.co/Zetv

A

Hey kubecon uh happy to be here, uh let's dive into our presentation.

B

Today, we're talking about managing applications in production and specifically using home versus ytt and cap to do so.

B

I am chaterupa. I work at vmware um and I have been working on deployment and packaging um for a few years now uh previously working uh related to cloud foundry and now on kubernetes. That's a picture of me in glasgow back when traveling internationally was still a thing.

A

And I'm dmitry, I also work with rupa on deployment group. I've been bouncing around in vm and container orchestration for a while uh happy to be here.

B

So in today's talk, we'll cover a few things, we'll talk about what we really value in deployment tools and what we're looking for there we'll see how a deployment workflow goes on kubernetes.

B

What are some challenges we faced with hum in regards to this deployment, workflow and then finally, we'll talk about some insights, we've learned from deploying in production at various scales um and talk about why we built ytt and cap based on our experience, managing these applications in production.

B

So what do we really want from deployment tools? Well from our experience, um both with large-scale production applications and smaller scale? These are really the things that we've come to value. So first is safety. We want to be able to catch mistakes before they happen and especially so in production.

B

We want reliability from our tools, so we want it to be predictable. We want to know what is going to happen is what we expect to happen.

B

We want transparency, so we want to know what's going on with the tool what's going on with our deployments, while it's going on and this kind of flows into the next point of debuggability, because we can use that information to fix things when things inevitably go wrong and then finally, we want speed. So we don't want the tools to be slow. We want to keep those fast feedback loops so that we can make changes.

B

Once we have these things, ultimately, we want the deployment tool to be boring. We want it to get out of our way so that we can go about and do the things we actually set out to do.

B

So helm is widely used in the community today, and so we looked at home because we saw lots of people having good experiences with it and we used it to deploy third-party software. We wanted the same thing to deploy our own applications. We wanted the same tools and workflows.

B

However, when we started using home, we faced various challenges, some due to the design choices that are made by hub and some concrete bugs that we encountered and we'll discuss both of these categories in this talk and describe. Why hum wasn't the right choice for us?

B

Let's talk a little bit about the end-to-end deployment workflow now in our experience, this can really be broken down into these four discrete steps and thinking about each of these as cooked kind of smaller jobs to be done, such as authoring configuration then taking that configuration and packaging and distributing it, then consuming that configuration customizing it to your environment and then finally deploying it and thinking about this in each of these steps makes it so that it's more transparent and we know what's going on in each step and it's safer.

B

So we can catch mistakes as they happen in every step, rather than learning at the very end that something went awry.

B

A helpful terminology that we found that we'll use throughout this presentation is thinking about the configuration author and configuration consumer and for third-party software. The author and consumer are different right, so we're consuming software that that is written by someone else. However, for software that we write and deploy to production ourselves, we play both of these roles and ideally, we would like the tools and workflows to be the same for both configuration, authors and consumers, and if we can think about these things, uh you know these four steps a separate jobs to be done.

B

Then we kind of started wondering. Can we create clean api boundaries between these so that consumers can pick and choose tools of their choice for each of these pieces and we'll discuss this approach more in this talk, but first, let's get started with uh authoring configuration, so the main problem we're trying to solve is we want to be able to provide some mechanism, such as a template to allow users deploying software to customize it to their needs.

B

Now, most templating tools out there, including home, uh are based on text templating and here's an example that you know is is using how many will look very familiar to many of you in the audience.

B

One drawback with text, templating and and in this example, is that I have to constantly think about constructing my own yamo and that is error prone, so to have a safer tool.

B

I'd like stronger guarantees that, whatever I write, always produces valid yamo and I don't have to worry about the right, indentation and coding values for vmware escaping I'd like the tool to be able to take on those problems. For me, another drawback with this approach here is that, if you think about configuration, knobs validations whatever else, while also making sure that I'm constructing valid yamo and intuitively, I want to think about the data and its structure separately and then think about serializing to yaml as a separate or a final step.

B

Now in this example, we're doing the same thing as what you saw above or before, but with ytt and you'll notice that I'm writing white valid yaml the entire time.

B

So I can focus on the values that I'm templating instead of thinking of constructing that yaml uh and then the data values uh pod annotations here is provided by a configuration consumer and it's associated with a particular yaml node by ytt.

B

Now, let's see an example that highlights the advantage of using data structures instead of text.

B

In this example, we want to build a config mac with a host port and a lowercase output, name uh notice in the config function at arrow1. I'm returning a map containing these key value pairs, not text.

B

This then enables me later to serialize this map to json when converting or when inserting it into the config map at arrow 3..

B

Since this is a simple example, I'm just directly taking this map and converting it to json, but you can imagine performing other more complex operations on the map too. The key point being that, since it's a map, I can modify it and transform it and take advantage of that structure instead of treating it as text.

B

Note that you don't always have to manually serialize all data structures. Next to arrow 2 you'll, see I'm using the output of the labels function and ytt will serialize the output of this function to yaml, for you.

B

Taking this concept of using functions and thinking data structures a little bit further, uh we realized that actually templating can be viewed as one big function. There are some parameters on which the templates run and there is a resulting structure.

B

You can either modify the structure further or if you're done, you can serialize it to yaml and then get the rendered configuration to make things easier for the configuration consumer. We have found it useful to have parameters or data values. Follow a particular schema.

B

Now, in our templating adventures, one thing we encountered often is that we were in situations where we wanted to install open source software or install third party software. In this case, we wanted to install a cni called calico.

B

The instructions on calico's website mentioned downloading calico type type, configuration yaml file and then changing the number of replicas based on your environment, and I could use customize overlay to do this, but I wanted something which we could do within this templating tool and following the paradigm that we shared before- and this is why ytt overlays are helpful. Overlays with ytt provide you with a way to patch existing yaml um and so you'll see on arrow one.

B

We get the calico typha yamo, and then we apply the yamo uh or apply the overlay on arrow two and then finally, an arrow three you'll see there's the change replicas, which is the over labor appliance. The chain replicas has also been generated within um ytt.

B

So now, let's talk a little bit about the execution environment of those functions that we're running prioritizing safety. We wanted to make sure that the environment did not have access to entropy network disk, the osa environment or time. This way the execution environment is always the same and ytt is reliable as a tool.

B

It also makes ytt stateless. Now uh you may wonder if you don't have access to these things. That means you cannot use. uh You cannot generate secrets on client-side or generate random strings and that's true. um We recommend generating secrets and storing any other state on the cluster itself.

B

And this is what all of this looks like in action. So if you were to use ytt, um you could have your templates in a config directory or any other directory of your choice and then apply values to it and then ytt would render the yaml for you, which you can then use with the deployment tool of your choice. In this case, we're using kukaro.

A

So let's talk about packaging and distribution. Now that we've figured out how to write our configuration.

A

So what is a bundle? uh There is no formal definition of a bundle. uh As far as we know, however, the way we look at it is uh there needs to be some kind of a bond between a set of configuration and a set of images that that configuration uses. Now, as all of you know, helm includes packaging component in its in its tool. uh People distribute charts over helm repositories.

A

uh You know, I think it's uh it's a tgz that gets put in the server somewhere more recently, though, helm has been encouraging people to use registry as the storage uh for helm charts right. So what you do is you take the the helm chart you run the command with helm and that helm chart suddenly becomes an image inside that registry right and ultimately, somebody else can pull it down and uh and use it right um now. Helm itself does not actually care about what images you're using within the configuration right.

A

So it's really just the top part of what helm is doing. It's dealing only with the configuration image for our own use cases. We actually need something more specific, because you know in a lot of our environments, we need to bring down the images that are being used by the configuration into customer registries uh to support cases like firewalled environments or air-gapped environments.

A

We need effectively a way to keep track of what are these app images that are being referenced by the configuration, and so we we see bundle as one thing that combines the two together. uh What's also important is that we want to retain uh for the bundle image uh the same digest throughout that way. The customer is absolutely sure that they're installing the software that they originally wanted to install even after it gets copied between various registries.

A

So how does this look well? This looks something like image package push with a set of files. uh This could be any set of files, could be ytt templates, plain yaml, maybe even a helm, chart source and ultimately, it's image package pool uh from you know a different location to download the bundle and maybe in between those two commands, you do end up using image package copy command to transfer the bundle from one repository to another.

A

Well, now that we're done with packaging and distribution, um customizing configuration is our next step. In our journey we've seen out there a lot of helm chart authors run into a problem that we call api mirroring over time. Consumers of the charts request to have various configuration, knobs be added to the helm, charts, and that becomes a big problem for the authors right, because various users may need to have various use cases satisfied and, ultimately, the set of values uh the template values grows, maybe even to thousands of options.

A

That's not something that we like and we don't think that's a scalable approach. So, let's, let's throw that out. Ideally, the way we look at solving this problem is saying that there is a percentage of configuration, that's very common to everyone right and that should be passed in as values to the templates right, so helm, values or ytt data values.

A

Once the template's done its job and returns the data structure back to us, we should be able to use a tool like customize or ytt overlay to apply certain set of changes to that yaml and finally, spear out the ammo for the following steps.

A

Ytt overlays is a feature of itt, so it's convenient to have the various uh functionality that you can do in your templates to be also done within the overlays. So we think combining the templating and overlaying approach is one tool is a very flexible and effective approach.

A

So this how overlay typically looks like it has a part about describing how to find a particular yaml document to modify within a stream of documents. So in this case it will find the single deployment.

A

Did you will raise an error if it finds more or less? You can tweak those settings, of course, and then it has a second uh separate section that defines what actually to change within that yaml deployment within the yaml document that's been found and in this case we're setting a priority class name too important.

A

So here's maybe a more complicated example. We needed for our dev environments to shrink their footprint, so we wanted to say find any of the deployment demon sets. Stateful sets if, if they have init containers go ahead and for each one of those containers, remove the resources section and remove the liveliness probe, resources, readiness probe and so on and so forth. So we have to cut this overlay short for this slide very convenient, and so this is how it ends up to be looking on a command line. You combine.

A

You extend that ytt invocation to include additional files and those files may include overlays and so you're able to specify your templates. Your data values your overlays all in one command. So that's uh that's it for customizing configuration. Let's take a look at deploying um deploying step ultimately boils down to taking configuration and putting it into a cluster.

A

We've had some challenges uh doing deploys in a repeatable manner, we've faced various errors and various cases that were surprised by, but of course, we also had successes, and so this is one of the examples. uh Example errors that we've seen sometimes occur. That was pretty hard to debug.

A

We've also had somewhat of a lack of confidence about the operations that will take place because helm was not quite transparent about the changes that I was going to make to the cluster, and even if there were options to uh you know tweak how the deploy would happen.

A

It was still not quite clear if this uh tweak like a force flag, would actually have some unintended consequences, so we thought well, there's got to be a more lightweight way to make us comfortable do the deploys, uh and so we wanted to write a tool uh that really focuses on safety, reliability and transparency, and so how do we actually see deploying to kubernetes work? Well, we wanted to build up on a basic kubernetes concept of labels, right, uh a collection of resources that has the same label.

A

uh We see it as a kubernetes application, uh nothing else really. On top of that, and of course, over time as you want, as you changing your application, some resources will be removed. Some updated some created, uh maybe tweaked in other ways, and so we wanted that to happen for all of our applications consistently.

A

So let's actually jump into a demo and take a look at how is it actually possible with this tool that we've made so here I'm going to type in capital s to list the applications on my cluster?

A

This cluster does not have currently any applications, so let's actually add one we're going to call it app and we're going to give it an initial yaml.

A

And so what we see on the screen over here is that cap will tell us before doing any kind of a change to the cluster. uh What's actually going to happen, we're going to create a deployment, we're going to create a service, and then we're also going to wait for those resources to reconcile. So, let's press yes, now at the same time, let's take a look at that configuration configuration is very simple: it's just a basic service and a basic deployment.

A

We do have this environment variable that we call hello message that has a particular value, we're using an existing docker image, so nothing out of ordinary. So a few seconds later cap tells us the deployment has reconciled right. So at some point it saw that the container was creating.

A

The pod eventually became ready and cap succeeded over here right. So this notion of having a clear log of what's happening to your cluster. uh While it's happening is very appealing to us. uh A summary of the changes that's about to happen before they happen also increases our confidence that we're actually making a good change uh and also even a convenient, tiny little line over here.

A

That tells us which cluster we're targeting gives us even uh gives us a little bit of a boost of yes, we're actually doing the right thing, we're not deleting production cluster. Maybe when we're when we meant to update a staging environment.

A

So let's actually take a look at this application.

A

It is, it is if you notice here six resources, even though we only deployed two resources right, so cap has capability based on those labels to actually track some of the resources that cluster creates right. As you can see over here, the owner, uh the owner column, specifies cap and cluster as different owners for different resources. So we cap only created deployment and service, uh so that's somewhat convenient.

A

We can now even see it in the tree view where some of the ownership references are being accounted for and so deployment owns a replica set owns a pod, and even for some of you who don't use some of the other external logging tools, it might be useful to be able to just say, cap logs dash f, dash, eight app and actually see a live log from all of the pods within that application. Of course, you can also use other tools because uh the pods are labeled.

A

uh You can use the other tools to to track the logs uh in bulk. um So let's uh go ahead on the second step over here and apply a change to the application. That's ultimately where we're here. For uh so I will go and throw in this dash c flag. uh It will show the changes that it's about to make has a nice colorful diff. It's telling us that we're going to create a new service, uh the service got renamed from simple app to simple app node, we're actually changing a deployment a little bit.

A

We have a value being switched to a config map reference. We have a config map that we're going to reference and then finally, we have uh the the deleted service. So let's go ahead and apply that and again we'll see the progressive log.

A

All right so before we jump into the next file over here, let's take a look over here for a second, so we have the config map and we have the config map reference. Now. Some software cannot reload configuration off of disk as it's running now, even in some cases. That might not be a good idea because you may not want to apply that configuration to the entire set of pods that you might be running for your service right, and so we have this feature in cap called uh versioned resources.

A

Really it makes the resource immutable. So any change to the config map will turn to be a new resource. So let's actually apply that and let's see what happens um so, we'll see in our summary that we're creating a new config map that has a suffix version one and we're also deleting the old config map, and we also have a change in the deployment because cap realizes that that config map is being referenced. So it will update that reference so that update will go ahead and complete and just to prove that this feature works.

A

When we change the config map, it shows you that hey we're going to create a new config map with version two: that's the change from the previous version right and then the version will be also used in the deployment correctly as well. Let's go ahead and apply that and we'll see that we'll see that the deployment kicks off now in helm. uh This functionality really uh is up to the user of templates.

A

The template arter has to do a little bit of manual bookkeeping. They have, to let's say, run the content of a config map through a digest and add that configure that annotation onto the pod. uh We think that a first class support for this kind of a feature is uh more in line uh with with how we use uh somewhere tooling here.

A

um So let's uh go ahead, and um so I've made an update to be yaml right, and so what is in 3a in 3a we're adding a new resource, a job, it's a dummy, little job, that's supposedly called. Does the db upgrade? That's not, of course happening in real life. All we're doing is sleeping for five seconds, so let's go ahead and do that. So we'll say: 3a we'll get our nice summary over here and then we'll apply that do note that immediately we actually see logs from this job in line within the deploy progress.

A

That again goes back to the transparency of what's happening in the deploy right, and this is because of the little annotation that we added over here telling cap hey. Please show this logs uh in line.

A

So, okay, we have this job. Now it's doing a database migration, let's go ahead and actually apply a change to it and we're going to change just a tiny little bit over here we're going to change the job to say from done to finished: let's go ahead and see what happens oof.

A

So a big error. This error is coming directly from kubernetes api. uh We actually want to format it uh in a better way right. We we try to pull out individual pieces in the list of the error uh reasons, and one of the reasons why I can actually see this is that we have spec template and spec template field is immutable right, so we cannot change it. So this is one of the interesting things about kubernetes.

A

Some resources may decide that something is not allowed to be changed right and there potentially some other way to to cope with that. Maybe to recreate a resource uh for port disruption, budget, for example, before kubernetes 115. That was the case couldn't update it had to delete create. So we actually have a feature that we think is more uh is more resilient to production usage.

A

We think that a user should have to annotate a particular resource and state that it is safe to recreate. So this is somewhat of a different decision point compared to just adding a four slack for all of the resource, potentially right, because who knows, maybe some resources will fail with an error and they may have to get recreated if the resource, for whatever reason is invalid.

A

So let's go ahead and apply that yamo with our annotations and note that actually cap tells us that hey there's a possibility that this will be replaced, and so we see our log again. We see that job is successfully updated, really recreated in this case, and it finishes great all right. So we've got, uh we've got one more example over here, so we have this job, but the problem is job is running at the same time as the deployment being updated right now for some applications. That's perfectly okay.

A

Some applications, though, may want to actually do a db migration before the application. Let's say, starts right now, there's various ways how to deal with this kind of thing. You may create a unique container, but we think a more first-class feature is deserved in this case. So what we're doing here actually is we're annotating job to be able to be scheduled as an update before the deployment and the deployment is also annotated to say, hey, I am the deployment right now. This is not specific to jobs and deployments. This could be applied to any resources.

A

So let's go ahead and actually try that out. So we'll we'll see what the diff is here. So one we're changing. Config map, which will trigger the deploy uh and, as you can see over here, deploy, is changing, we're also changing the job, so it will actually run. So, let's see what happens so, as you can see, we got a job running over here.

A

We've got some logs coming out, we're waiting, we're waiting for it to succeed, okay, it succeeded and then, finally, after it succeeded, we actually have a deployment update running and then it finishes.

A

So the ordering here is really not a special feature to cap. It is quite similar to namespace and crd features uh within cap ordering of those uh it's the same system. Cab just exposes and lets you use it yourself. We don't treat crds, for example, any different from any other resource, so happy to deploy it and as an example.

A

If we take a real out there app assert manager, we can actually see how it all works out. uh So here's a few resources there's a single yaml file. You don't have to split your cds or whatnot, and you just let it go and cap will go ahead and deploy uh cert manager and uh in a second or two, all the pods should come up and the cap will tell you insert manager is installed there. It is all right. So, let's switch back to our presentation.

B

So with the clean api boundaries and small building blocks, uh this is all beginning to to look good. But, ultimately we said the goal of good deployment tools is to make it boring to get out of your way. And how can we get there? How can we create a simple deployment? Experience we'll see some examples here.

B

So this is uh some client-side deployment examples, if you're using clis and in the first example we use ytt for templating and then um cutoff for for deploying it in the second example, we use all of the tools that we've showcased here, so we use image package to pull down a bundle and then use ytt to template it with the appropriate data values and then use cap to deploy, and then in the final example, we continue to use home to template, maybe some third-party software that is shipped with home templates and then use cap to deploy, and so with this, you can kind of see how, as we've talked about before it really allows you to break down that deployment into these different pieces.

B

uh Giving you a little bit more uh visibility into what's going on at each piece.

B

And cli, based experience, as I showed before, is, is helpful, but really what we want is for this to happen behind the scenes almost in a boring way. So, ideally you want this happening on your cluster. You want some kind of continuous delivery mechanism, especially if you're running in production and that's exactly what we have with this tool called cap controller. So cap controller provides an app crd and that's what's here on the screen, and it tells you very clearly the three various things it's doing.

B

So what is it going to use for the fetch stage? What strategy is it going to use and it supports various strategies? What is it going to use to template and again different strategies there and then? Finally, what is it going to use to deploy in this case cap? So in the fetch stage, right with various strategies here, we can either use image package to get a bundle.

B

We could use home fetch to go fetch some home charts uh or you can use uh back it with a git repo, and so this is where you can kind of use it to do. Githubs, like workflow and any changes to the exit repo cap controller will watch on and make changes appropriately.

B

Similarly, for the template, we're using ytt templates here, but you could also use home tablet.

B

So, what's next with these tools, uh if what we presented here, these ideas they resonated with you, uh you can learn more on k14s io, uh try to use cap or ytt on get ytt, I o get capio.

B

We are always available and chatty on the k14s slack channel on kubernetes. So come talk with us share, share how you template and deploy today.

B

All of these projects that we shared are open source and the rest of the k-14s. Tooling is open source as well, and we uh welcome contributions to any of these projects uh and we love hearing uh how you use these tools. uh It really makes us happy so to come, chat with us. We hope you found this talk valuable and thank you for your time.