Cloud Native Computing Foundation GitOpsCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Are Your Declarative Configurations Human Friendly? Closer Look at Code Reviews with... Ryota Sawada

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Are Your Declarative Configurations Human Friendly? Closer Look at Code Reviews with GitOps - Ryota Sawada, UPSIDER

The great values of GitOps such as commit history for change audit and code reviews / approvals are underpinned by declarative configurations. Both Argo CD and Flux support not just simple Kubernetes YAML definitions, but also Helm Charts and Kustomize. They allow templating and some basic computation for generating manifests, but how declarative are they "to humans"? Any changes to a Git repository can have a significant impact on GitOps platform setup, and code reviews play a significant role for ensuring correctness. Even if changes are carefully made and reviewed, you can still end up with unwanted changes when templates such as Helm and/or Kustomize are used. They are great at managing large and complex configurations, but can be confusing or unclear to humans. This talk takes a closer look at the common pitfalls of using Helm and Kustomize with GitOps, how to tackle them using additional tools, and a potential alternative approach using a new tool called "Importer".

A

Thanks for joining welcome to get off scone north america 2021, I'm ryota sawada, this is: are your declarative configurations, human friendly, closer look at code reviews with gitobs, so let's get right into the agenda.

A

I'll start with a brief introduction myself and the company upside up and then we'll touch on the target audience and the goal of this talk. After a quick brush up on helm and customize, we will look at some example: pull requests with them. We will then take a look at this brand new project. I have been working on importer I'll briefly talk about the importer details and we'll go straight into demos.

A

Finally, I'll touch on other tools and considerations for great code review experience with githubs. So a little bit about me, I'm the cto at outsider, leading platform team and also responsible for back-end development. I have worked in finance space for about a decade now you can find me at usual places with ryt swd.

A

That's my full name without vowels, I'm based in london and have two cats. As you can see in the picture to the right, you can find that picture used as my profile picture in most places.

A

Also I'd like to briefly touch on the company. I work for upsider. It is a fast growing fintech startup providing b2b payment services. We are cncf, end user supporter and use git ops extensively in production.

A

The headquarter is in tokyo, japan, but the remote team is around the world. As I myself I'm based in london. Before moving on to the actual talk, I have to touch on this must-have topic. We are hiring if you are interested or curious to know more, please feel free to reach me at rit swd.

A

There are mainly two materials. This slide, which is made available via tiny url shown here, and the repository used for demos under my github profile, rytswd, slash, githubscon-na-2021,.

A

This talk specifically focuses on code review with githubs as a prerequisite.

A

It is expected that you're familiar with the github solutions and also use of helm and or customized with githubs for the target audience if you review code for github setup day to day, are looking for better code review practice or just had enough of helm charge syntax. Hopefully this is the talk for you.

A

This is probably not for you if you have no issues or whatsoever with code review with helm, customize and or other tools for classes set up at the moment.

A

By the end of this talk, hopefully you will familiarize yourself with some of the items mentioned here, such as understanding how git ops requires extra care for code, reviewers, tooling, to help herman customize setup. Very first look at importer and more, I know we all love code review, and this is probably clear to everyone. But let's touch on this briefly code review is done to ensure code correctness and get explicit sign off from reviewers in case of git ops. This would be the sign off to actually deploy something to clusters.

A

There are several tools and ways to review code, but for this talk I assume we are reviewing code from the web ui and we are only talking about github's code. I think it is safe to say that reviewers should have a good understanding of kubernetes resources, but what about the other tools? Are they truly required if a new configuration management tool emerges? Does that mean reviewers need to learn the new syntax, best practices quirks and everything else by the tool? The more tools you have, the more difficult it gets to review correctly?

A

We will check out some example: pull requests in a bit, but let's have a quick brush up on helmand customize. First, so helm is about its chart and the key files are, as shown here. Helm charts can be quite complex and from the code review perspective, go template. Language for templates can be difficult to understand and review.

A

Customize about customization.yaml files to define the list of files, you need and use those customization.yaml files with base and overlays to generate a complete spec thanks to its patching capability. Environmental differences can be handled easily, like the structure shown here, but because of the nature of patching it can be problematic for code reviewers as well.

A

So now it's the first demo time. Let's go right into my repository and check out some example for requests. I have created each pull request, has detailed description from overview and challenges to potential solutions.

A

I won't be touching on every point in this talk, as this is more for your reference later on. So let's look at this first one. It's about upgrading grafana from 6.3.0 to 6.16.2.

A

These versions are really chosen, but let's have a look at the file changes, so we can see that the versions do match up. So we are updating from 6.3.0 to 6.16.2.

A

But that's as much as we get, there is no clarity about what kind of resources are being created, removed or updated from this poor question file changes so for this in order to actually see the resource changes, you need some toolings to help on this particular example. I have set up this repository with slash, run, div automation, so this automation uses this dyff cli tool to have the div between two yama file inputs.

A

Those two yaml file inputs come from this helm, template commands being run behind the scenes, so helm template is run against 6.3.0 helm, chart and also 6.16.2 helm chart. So if you look at the result, slash run diff comment like this.

A

You can see the diff of 50 differences before corresponds to 6.3.0 and after corresponds to 6.16.2, and you can see 50 differences. How exactly each change would result in the cluster different, the resource differences and you get another one. This is for staging and you can see how the differences are being made for each environment differently. You may have values differently.

A

For that reason, you need two differences or multiple differences based on environments that you need to deploy the helm charts against too.

A

So. The next example is updating some boolean flags for helm chart and this one uses the flux helm chart. I have copied the flux's helm chart here in this repository. You shouldn't be doing this. This is just for demonstration purpose. If you are to really use flux's helm chart, you should be using directly or by dependency.

A

If you look at the file changes, these are some changes of enabled flag to false to true secret config, so on and so forth again this time you don't really see what actual resources are being created other than you are updating something related to prometheus, something related to secrets, etc.

A

So if we run the run diff command once again, it does give me that it's only updating staging, so it was originally meant to be only for staging and we did see only one file change, so production is not updated, but for staging we don't see any output. This is a limitation with dyff. Eyff can only handle if the two file, two yaml files have the same number of documents.

A

If you update some of the enabled flag, it is likely that some new resources are being created or some resources are being deleted, and for that reason the document number don't match up with the two files.

A

That means, if you are to have some automation and tooling, around helm charts this dyff automation might be too simple to really help you code review more efficiently.

A

Let's look at another example this time with customize, so this is about updating some version for one deployment. So if you look at the file change, we can see that toolkit alpine. This particular image is updated from zero point two point x to zero point. Two point y: just one line change one letter change- and this is very straightforward base update.

A

So this is to update all the environments, production and staging, and if there is any other environments, you would probably update all of them and if we run the run div once again for this particular change, slash run diff. It actually gives me that for production there is no difference for staging. We can see that the 0.2 point x is now being updated to 0.2 point y.

A

So that's expected staging is working as expected, but production there is some overlay or some patching happening, or maybe resources is not pulled in for production, so for customize it might be simple enough, but from the div point of view from the code review point of view. Actually it doesn't tell me everything unless you have some automation or you go through all the code or the customization.yaml files.

A

The next one with customize is about adding some missing service account. So if I look at the file change, some service accounts are being added. Customization.Yaml is listing another file service account.yaml, which corresponds to this new file. It's adding this color service service account, but there is another one. That's created service account. Termi.

A

Is this an ok file? Is this something that was missed in the customization.yaml? Or is this just for some later on usages? This is unclear from the code ready, perfect review perspective. Customization does give you this flexibility, but it does give you less clarity why that file is needed. If I run the slash run diff once again.

A

This is another limitation that we hit with helm the number of resources change between before and after the change, and for that reason diff cannot be shown and for this kind of dish to be shown, you need to invest in some other automation and some tooling to help you do the diff, maybe unique, stiff or sniff might be able to help, but this is something that you need to invest in.

A

So we looked at code review with helm, helm is a package manager, so working with a third party tool is very nice, but, as we saw in the simple examples just now home template commands may be required for reviewers to understand the actual impact. We didn't have an example of template files being updated, but I think it's evident how that can be also tricky to review the examples with customize looked straightforward, but it also had some unfriendly behavior to reviewers the code. Change didn't explain how one change affects another one environment, but not the other.

A

In order to get to the bottom of the spec, one would need to check all the relevant customization.yaml files if the diff cannot be shown, such as the case we saw at the end, it gets even more difficult, so we want better code review experience. After all, it needs to be easy to review.

A

Files should be declarative, just like simple kubernetes are more spec, but still provide some sort of code. Reuse and the syntax must be simple. Also tooling needs to be simple, easy to install no complicated logic when used and can be easily integrated with ci and any automation setup.

A

Now it's the time to introduce importer.

A

Import is a cli tool to import any lines from anywhere it works by passing the provided file and look for special markers, which are mainly provided as language comment. These markers can define where to import the code from as you can see, on the right hand, side here, the imported lines appear in the file. This is an example of generating a new file and we will look at other examples in the demo later importer is a brand new project. I have started. You can find some details here and also full details and background.

A

In importer repository importer is designed to be simple and in a way dumb, it doesn't know anything about filed validity. It just looks for matching markers and import some data when necessary. This design was the key to ensure it can work with helm and customize importer is not to replace the existing tools but to augment them. When you find code review, getting more and more difficult, importer may be an option worth looking into.

A

So now it's another demo time so now, let's see important in action. First of all, how do you install importer? It's really straightforward through install upsider tap importer, if you want other installation option, all the details are provided in the repository, because I have imported installed already. Let's see that in action, I have some files prepared here, markdown demo markdown, so this markdown has some data, and this comment is the special importer marker syntax, and there is another one down here: importer marker is always a pair.

A

There is one for begin and one for end, and this one is named as a short description and it tries to pull in the data from this dot mac this markdown snippet.md, with a symbol of full demo. Let's see what this file contains.

A

So this is another markdown uh with the description, some other content, and you can see another comment. Export for demo begin export for demo end. This is another marker of importer provides importer uses, and this is an exporter marker. So this for demo is the symbol that that matches up with the importer marker and the data down here, we'll be replacing the data up here within this importer marker pair.

A

So let's see that in action, importer update, markdown demo now see the file, so you can see that the file is now updated. It imported the file and the content from markdown snippet.md.

A

Let's see another one, yamo demo yaml so this time it has some data description and it has some data outside of it. Where we have this comment, another comment with important marker important marker is a pair again. It's begin and it's got end and this time I'm looking at yamo sniper, yamo and metadata only symbol. There is another option for indentation making sure that it will be aligned to the two spaces. Here.

A

If I look at yamo snippet.

A

He has some other data named description and within the test data stanza, you can see the exporter marker appear twice. One is for sample nested, which is begin here and end here, and there is another one that is nested metadata, only exporter marker, which is ended here, and it has metadata name and namespace.

A

Let's see how that gets, updated, import and mark that update, yamo demo and see the file, you can see that now metadata, name and namespace. This data is pulled in from yaml snippet and make make sure that the indentation is correctly two spaces for this case. If we just validate by looking at snippet, you can see that the indentation was different from the original snippet file.

A

Now, let's look at a pull request based on importer.

A

So this one is updating our cd from version 2.1.0 to 2.1.2. So if you go directly into file changes, we can see an important marker at the very top. It says it's got the marker name of our install. It begins here and from a file of github.com our approach august cd 2.1.0.

A

This is an official installation, spec algo provides and I'm updating that to 2.1.2, because importer update has taken place already for this file. You can see that 2.1.0 is now being updated to 2.1.2, for some of the image tags and also what's interesting, is that you can see additional m variable defined for 2.1.2.

A

This is something that's very clear and because importer has updated all the files, this makes the file completely declarative.

A

If we go back to description, we can see details and benefits, but those are covered in the slides. So, let's get back to the slides, so with importer you can clearly see the resource div. It also means that any resource additions or deletions can be fully tracked. By commit history.

A

There is no hidden dependencies or external factors with importer and all the files become truly decorative for each file. Customize helps declarative setup, but it doesn't guarantee to have each file to be fully declarative. There could be some patching on top of them. Helm uses heavy templating. So, although helm values files are declarative, helm, template files are not declarative in most cases.

A

So, let's quickly summarize, the importer takeaways first of all importer, does not aim to solve everything. Importer takes code generation approach and for that reason the code can get stale easily. You can fix that by adding some automation, which you can find example in the important repository importer also does not have any template, repeat, etc. Things listed here these may change in future, but it's kept as simple as it can be for now.

A

In version 0.1.0, there are plans to have ide support, nested important marker handling and more in the future, but it's a brand new project. After all, I'd love to hear your feedback and ideas before wrapping up. I need to mention about validation. Validation is beyond this talk because you can end up with valid but unintentional definitions.

A

We looked at some examples, for example, pull requests with helm and customize, and they were all valid from kubernetes resource point of view, even if you had some custom validation for you, your use case in the ci pipeline, some unintentional change can still creep in from the code review perspective. Valid does not mean that all the changes are good to go as to the validation itself. There are great tools out there, such as oper, good file, etc, which are definitely worth investing in for better pro requests and merge.

A

Request management here is a couple more tools that could provide better code review experience, hcl, hashcorp configuration language and queue are both great tools, but, as mentioned earlier in this talk, it is important to take code review experience into account before introducing new tool.

A

Finally, here is the list of tools and items I used in this talk, although I touched on dyff devs limitation, it's a great tool and definitely worth checking out. If you haven't done so already.

A

There are some other tools, such as chat, ops, automation, setup, which you can find references here with that thanks very much for listening feel free to reach me on twitter. If you have any questions or comments, thank you.