Cloud Native Computing Foundation KCD Berlin 2022, 16 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Stop being YAML Engineer, be a Software Engineer with CDK8s - Robert Hoffmann, AWS

Description

For Kubernetes, there are plenty of tools to manage and scaffold resource configurations. One thing is common to most of these tools: They use some kind of configuration language, like YAML.

Website: https://aws.amazon.com/de/

Organized by @Microsoft @kubermatic7173 @SysEleven

Thanks to our sponsors @CapgeminiGlobal, @gardenio, @sysdig, @SUSE, @anynines, @redhat, nginx, serve-u

A

I want to know awesome, yaml Engineers here in the room, I would bet yeah I, see some hands, definitely I think to some extent we are all yamla engineers in this world right, it's not just about kubernetes, but especially about kubernetes and lots of other tools that let us define configuration in yaml I have only one type of yaml I personally like and that's the yaml I do not have to write, and that's the main part of my talk today.

A

So I will show you a way where you can, instead of writing, yaml use programming languages to configure your kubernetes applications, but, first to remember the joy of writing yaml to deploy kubernetes apps. Let's have a simple example to basically remember to remind ourselves so um in this example, we're just writing a little bit of kubernetes manifest for a web server. That just gives back some response, so basically in HTTP Echo server- and you probably have seen this one million times, there's like at least three things.

A

You typically need to do to have a web app. That is also Exposed on your cluster, so that it is reachable. So, first of all, uh you write a deployment uh object there you specify at least the image of the container you want to run, and you also specify some labels and some ports now.

A

The second thing you tend to do typically, is you have to define a service object and that service object is basically giving you a stable endpoint within your cluster, where you could then reach that deployment and its parts behind it. So again we have to Define some things here, so we have to put in some names. We have to again write the same port down because that's the part the application has been listening to what we have defined in the deployment.

A

We need to basically write that again in the service and finally, we also have to Define an Ingress and that's our way of saying. Okay. We want this application to be reachable outside of our cluster and there are many Ingress implementations that make that happen for you. So again we have to write a bundle of yaml and again we have to reuse some of the identifiers numbers things we have already declared before.

A

So it's a little bit repetitive, so just to give you some example here we you see that we need to Define some labels, because labels are the way that those kubernetes objects find each other to work off of another. So, for example, we have this app Echo key value here and we're actually using that later in the service to then select all the ports that should be behind that service.

A

Now same thing for the container Port. You specify this in the deployment we have to do it again in the service, it's pretty repetitive, and then we also have a repetition here in the Ingress controller, this could be different, but again, there's there's a pair of values. uh You have to basically copy and paste around.

A

Finally, we also have uh to name things and naming typically can be hard. So here we have to name our service and we have to make sure that the name isn't already taken and then we have to again copy paste that name into our Ingress, because the Ingress needs to know the name of the service and in total for just a very simple web app, that's reachable from outside my cluster, we've wrote 45 lines of yaml and I.

A

Guess that's one of the reasons why if you actually have a sophisticated kubernetes application, you're, potentially talking about hundreds or thousands of line of yaml- and this is not even just true for kubernetes- it's something I- think you experience with a lot of infrastructure as code things. Could it could be cloud formation could be terraform, it's not just about kubernetes.

A

So, as I said, um this has some downside. So we wrote this in yaml. uh It's it's like a static configuration language. We don't have like things like variables out of a box. We could use some templating tool to do that. Actually we saw that there's some copy paste, so either I copy the value around or I copy my placeholder value around.

A

So it's not much better and it can be hard to customize and share again I, probably have to put in a lot of placeholder variables and use a templating tool to make that happen um now. An interesting part about that is that it also carries a pretty high cognitive load and what I mean by that is um maybe you've seen colleagues build, kubernetes manifests and they, you know, do a pull request and you do a review and you get like a yaml file with 500 lines that have been changed or 100 lines have been changed.

A

So what do you do? You write lgtm and you emerge it right, um because there's no way you can actually understand what's happening there, all these objects that have been changed in that pull, request, they're kind of interconnected, but you basically have to write it on a wall and draw the lines between the ideas. Oh, that ID is used in that object in this object. So this belongs to that and you don't really know anymore. If you just look at that wall of yemma, so that's what we mean with high cognitive load.

A

Now, let's talk about a tool that might be able to help you here, and this is why I introduce cdkates. So what cdkates um first to explain what silicates is?

A

We have to talk about the motivation we already established that authoring, those manifests can be a daunting task and what cdk tries to do is making that authoring simpler and also more robust, basically tackling the challenges we had in my example and how it does that is by leveraging general purpose programming languages where you can then apply standard software engineering processes instead of using yaml as the main language to describe your deployments.

A

So to give you the formal description of what cdkates does it's an open source, multi-language software development framework for modeling kubernetes resources as reusable components, and it does that by giving you the option to use one of five different programming languages right now you see that at the bottom of the slide, so you can do it in Python, JavaScript, typescript, Java and in golang, and instead of writing, yamu you're, actually using one of those programming languages.

A

Probably your favorite one to Define your kubernetes objects and then cdkates will turn that into the yaml files and it's that type of yaml file I like because I didn't write it cdks did write a form. I can just read it and verify it.

A

um So that of course gives us the ability to cut down on a lot of that copy paste we saw before, and um it also makes it so that you can easily share uh your work uh in basically building your own classes, where you compose those objects together and share that maybe in your company or publicly and the sharing doesn't work with some special tool.

A

It's the way you would share a library in your favorite language, so you would use npm if it's a typescript or JavaScript thingy or you would do Maven or whatever, depending on your language, so there's basically no new stuff. You need to learn or use it's basically the tool chain of your favorite programming languages. You already know and love, hopefully,.

A

um So what does that give us when we use programming languages instead of yaml? So first of all, it's Dynamic and some people might not like that, because it's not you know fully declarative anymore. You have like control flow, so you can do loops.

A

You can do conditions whatever, but if we're being honest, if you look at things like terraform and other languages, those configuration languages become more and more Dynamic right, because you need to solve those cases where you have inputs that change the way you deploy the template and it gets more and more complex. So the let's say the gap between being a configuration language and being a programming languages gets narrower and narrower by day.

A

So at some point you could ask yourself: why not use a programming language to begin with, and that's basically the idea behind CD case what we can also do now that we're doing coding is, of course we can follow the do, not repeat yourself a principle or dry, so we can pull out certain variables certain values. We can reuse them, so we don't have to copy paste them all around.

A

Finally, I also talked about the standard package manager, so you can package your stuff and maybe blueprints for your internal use or publicly just with whatever the package manager for your favorite programming languages and then, of course, because we're writing software. We can use abstractions. So uh in the end, you do not have to write out every single object.

A

You could easily think how you could compose these objects together to make a new class that you know is basically a template, but now it's in a programming language- and you can share it with your friends or in your company, and we will look at that a little bit more um as I go along now. I want to give you a quick dive into how it works, how you would actually work with cdks and how it feels so so you have an idea on how to use the tool.

A

So, first of all, uh you need the cdk CLI, so there's a CLI tool and there are various ways to install it. For example, through Homebrew, once you have the CLI, you init your project and for initializing your project, um the command line gives you a helpful command called init, obviously, and there you can put in argument for basically your favorite language, and this will make the see the cdkates CLI to basically scaffold a basic project for you.

A

So it's it's already ready to go for you to actually write your kubernetes objects and what you get is basically what you see here on the right side. So this is, for example, um when you initialize it as a typescript app you uh get a basic setup with a package.json. You already get your like main entry file where you write your stuff um and it would already work and compile it just missing your objects that you want to add. There.

A

Now the next thing you do is you actually import um your API objects or the API objects of kubernetes into your project. So what does that mean? There's an import command and after that import command you can either specify to basically import the standard, kubernetes API with all its objects, and you can say which version you take. If you just do a cdk import without arguments, it just takes the latest kubernetes.

A

So it could be one point to something: I, don't know, I think this changes every day and basically, what it does is it takes the API specification and automatically transforms it into code for you. So after running that command, you have all the API objects of kubernetes, all the standard objects as classes that you can instantiate and they're strongly typed, with all their options and with the documentation, and we will see that in a second. So typically, this is one of the first things you do. You pick your kubernetes version, so maybe you're cutting edge.

A

So you have some one point to something, or it's a little bit earlier, so you take whatever version you're actually running and you can pull down all these uh API objects automatically. Now a cool thing is that you see here there's also a different way to use it as an argument. You can actually um pull in a custom resource definition file.

A

So if you are using any custom resources in your kubernetes cluster, you can provide the yaml and the CLI will also turn all those custom resources into API objects into classes with all their annotations and so on. So it's not that it's just about kubernetes core objects. It's basically about anything that is defined as a as a custom resource, and this is what you get when you do that, so there's an import folder and here, for example, you see uh the files it generated for the standard. Kubernetes API objects.

A

Now, basically now your project is completely set up and you just need to start coding um and we will look at how this works just in a second.

A

um What's really nice about this experience is that um if I show you a uh snippet from my editor, so I'm using visual studio code, you see that here um now my deployment is not a yaml file. My deployment is a class and I'm instantiating that it needs some attributes that are not too important right now. So this is like a context, object, and this is uh basically a um a logical name. You provide and then, when I start, for example, write the spec for this deployment.

A

You see that I get a complete code completion, because I created or I put in I imported all these definitions as typescript interfaces by running that import command before so I know exactly what I could write there and if I go further. For example, here I selected, the selector as a property field, I would now want to write. You see what's actually happening here.

A

I get the complete schema and I get the documentation of what that Shield actually does and I get this all out of the box, because it's just code and it basically Works in every IDE, as you would expect.

A

Now, let's have a look at how this actually works when we write the echo server, that I showed you before with a programming language and I'm using typescript here, because I'm, a typescript guy, don't judge me if you like something else feel free to repeat it with whatever programming language you like so, first of all, again we have to Define our deployment, but now it's that class that we saw before we still have to provide more or less the same input as before, but we can already be a little bit more dry because, for example, we can declare the port as a variable at the top, and then we can use it all throughout our code instead of repeating it copy pasting it around same for the labels.

A

We can just you know, Define our key value for the label and also pass that around now. We're writing the service, and here we can already make use of that. So you see instead of putting the value there we're using the port constant that we declared before and for the labels. We use our label variable that we're referencing here, so we already improved it a little bit now. Finally, we're also defining the Ingress and again we can make use of that Port variable we had before.

A

um So. By pulling this out, it already got a little bit better and you see here all the basically variables we could reuse now an interesting thing where it actually gets even a little bit better is um naming because what you saw here is um you basically didn't name that service.

A

You didn't have to think about an ID That's Unique in your cluster, in your namespace, whatever you just created a new service now you know the Ingress needs the service name, so you just reference it, but you actually don't really care what the name is that cdk is doing that for you, so behind the scenes cdkates make sure that all these objects are uniquely named and you can just rely on that and and pass references around so that helps you to be. You know much clearer about okay.

A

What object belongs to what now we've written our code um and the next thing we do is actually we synthesize it. So there's another command in the cdks library it's called Uh cdk CLI, that's called synth and with synth. um You are using that code and the syndicates CLI turns this into yaml for you. So this is what you actually deploy.

A

um So this is not Reinventing the Wheel by you know talking directly to the kubernetes API or anything. In the end, the output is a completely valid kubernetes yaml file. You can look at it. You can reason about if cdks is doing the right thing for you, but it's nothing. It's nothing. Really special and the cool thing is you just have to read it and not write it, and that's my my favorite yaml that you just have to read and then, when you have that manifest, um this is actually written out to that dist folder.

A

That's also part of the project. Scaffolding that you have set up. You can do with that yaml file whatever so you can. You know directly. Cube cuddle uh apply it that shouldn't be deployed. Should be applied, sorry for that should be Cube cuddle. You can do Cube cuddle apply. You can put it in any other git Ops, whatever tool that is able to work with kubernetes manifest, and you just you just deploy it or put it in that tool, and then it runs in your cluster.

A

So, basically, at this point here, that's really standard kubernetes operation has nothing to do with the tool anymore. The tool is just here to help you author. Those manifests to generate them for you now, as I already said, um once you did that you might have the idea that hey what I built there is actually super awesome and the whole world should use that. So, let's imagine that this HTTP Echo server that I set up I think this is like super valuable and will probably get 10 000 stars on GitHub, so I should release it.

A

So you could think that you are building your own class. Let's call it the echo server deployment, and this thing maybe even doesn't need any options, because I'm very opinionated on how it should be configured and in that class it could also do those three things you just saw with a deployment with a service and with the Ingress and then I can package this um for my favorite language and, for example, publish it to npm, and then everybody can use it.

A

And of course you could also do this just internally at your company, so you build a class that represents all the standards you apply for deploying in kubernetes application in your company. Maybe you have a special Ingress controller. You probably have to do like 1 000 special annotations to actually run something, so you could code this in as well, and then you for the users- it's just really easy to to just instantiate their class, and they get all of that great download.

A

Now, at this point you might be saying: okay, that's that's! Pretty cool I get some benefits from using a programming language, but uh in the end, I still need to know a lot of stuff about kubernetes and, if we're honest, it's just a different to some extent. It's just a different representation. I had to still I still need to know or need to fill out the structure of all those objects.

A

So can we do this even even better, um and this is where the cons concept of the so-called constructs comes in, so um the construct is basically just the class that represents your kubernetes resource um and we um Define them in several levels. So um and- and the level basically says how smart the class is, that you use so the very Base Class is the API object that we see here, and this is basically just a container for the yaml. So it's just like okay, every yaml resource needs like a version. It has a type.

A

It has a spec. Something like that. So it's very basic the next one- and this is the one you get when you do- that import command um are basically all the kubernetes core objects that you use on your day-to-day basis and that's automatically generated when you use that import command.

A

uh It's just a one-to-one representation of these API objects, but you already are in code, so so that's a plus, as we've seen now it really gets interesting is if we go to level two so level, two are actually handcrafted classes and apis that you can use to even simplify uh your configuration even more, and this is what cdkates plus does so just see the kids, but with a plus and what it does is essentially this.

A

This is the same HTTP Echo server that we saw before with like 45 lines of yaml um and now we're just down to I think about 30 lines of code. To do that so what's happening here is we're not using the standard deployment object, which is the one-to-one representation, but we're using the K plus deployment and the K plus deployment is a little bit opinionated. So we have to pass less arguments to it right. We don't have to think about labels.

A

The deployment already knows that um and then there's a convenience function which is called expose via Ingress, and that does exactly what you would think it does. It exposes the whole thing for you, as Ingress IT wires up the service. It makes the service naming all of that stuff we had to do before is like a boilerplate. We don't have to do any more.

A

This is done for us behind that function, so this makes it much shorter, so we're now just down to 13 lines of code, so that's less than three times over three times less than what we had before when we did it in Yammer and the really interesting part, if you think about collaboration, is actually that we are not declaring yaml anymore, but now through the power of programming and our ability to abstract things, we can actually declare our intent so, instead of just scrolling through a wall of services and ingresses- and you don't know, what's it all about- you now can read in the code.

A

Okay, somebody wants to expose that deployment through Ingress and you could imagine that you can drive this further by creating your own constructs your own classes that give other convenience functions that are meaningful for your business or your way of how deploying of how you're deploying kubernetes applications.

A

So that's a pretty pretty quick overview of what cdkates and cdkates plus do for you um I hope your your um found. Some excitement about the way you could use uh programming languages instead of yaml um I want to leave you here with some interesting resources that you might want to check out if you want to go deeper into cdkates.

A

So, first of all, there's the project website, cdkates, IO um and then what's really cool is um I talked about hey, you can publish your own packages with your own constructs, there's actually an open source community and an open source website where you can actually find all those things.

A

So there's the construct plus Dev uh website, that's the constructs Hub and there you find a nice search interface where you can actually search for cdkates libraries that other people have provided, and there are quite a few like super cool um extensions like things where you can automate deletion of resources that you remove from your stack and so on. So there's quite a few things the community already did and you could do as well.

A

um So while this project see decades has started in AWS as a lab project in in the meantime, it has graduated to be fully open source and it's even a cncf Sandbox project. So you can go here to actually check it out and we have an open, Slack Channel at cdk.dev, where you can interact with community and find more about uh more out about syndicates.

A

And finally, we have a really cool um Community event that happens once per year. That's the cdk day and that cdk day is actually about the whole cdk ecosystem. So cdkates is just one tool in the cdk ecosystem, there's also the AWS cdk, which basically does the same, but for AWS resources and then there's also the cdk for terraform, which does the same but for terraform. So you're writing code to generate a terraform manifests and you can learn all about that on the cdk day and I think the sessions are recorded on YouTube.

A

uh So you will find your way around um and with that I urge you to stop being yaml engineers and maybe think about becoming a software engineer with cdkates. So right now, I think the project has like 3 000 stars and it's still growing and would be really happy um if you would try it out and also give me feedback. So you can find me uh on Twitter at drop off Max or probably in LinkedIn. Just search for my name. Robert, Hoffman and I would be really interested in in your feedback.

A

If you try it out someday.

A