Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tutorial: Zero to Operator in 90 Minutes! - Solly Ross, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Tutorial: Zero to Operator in 90 Minutes! - Solly Ross, Google (Limited Available Seating; First-Come, First-Served Basis)

Please bring your laptop fully charged as we will have limited charging stations available in the room. Please complete the following steps ahead of time to make your tutorial easier: https://gist.github.com/DirectXMan12/ad7b35327c2816125a45cdc11ff78476 Come learn how to quickly get off the ground running with building an operator using KubeBuilder v2! Come write a Kubernetes-style API to manage a bespoke application, complete with declarative validation and defaulting. Discover what kind of requirements go into an API type, and how to write API types that work and feel like they're part of Kubernetes, and can be easily consumed as part of a larger system. Once you've got an API type, you'll make use of the new server-side apply functionality to make implementing your core logic a breeze, and learn how to think about writing well-behaved controller logic that deals with different interactions with other parts of Kubernetes. Finally, you'll learn how to actually run your controller locally for development and on a remote cluster for production.

https://sched.co/Uaeh

A

All right so before we get started, if you are having issues with like setting up your own cluster or if the conference Wi-Fi is being frustrating, please raise your hand. My coworker over here will come along with a GCP account information, and you can use that GCP count to create VMs and run some stuff in VMs. So I'll talk a little bit more about doing that in a bit.

A

It's still a little quiet.

A

A

All right so welcome to zero to operator. Thank you all for coming today. For those of you who don't know me: I'm Solly, Ross, I'm, a software engineer on G ke and a maintainer of cube builder, and my mission is to make writing kubernetes extensions, less arcane.

A

So before we get started, let's quickly talk about some definitions so to define an operator we have to define a controller. So a controller is a loop that reads: desired. State observed cluster state and potentially some external state, so things like load balancers on your cloud, for instance, and then reconciles the desired state with the observed cluster States and the observed external state and writes down observations at the end, so that other things can see what we're doing.

A

This is a relatively uncontroversial definition all of kubernetes functions on this model, everything from the deployment controller down to the node, even the scheduler. Ok, so now that we have that in place, let's define an operator an operator is a controller that has specific knowledge of an application.

A

So, for instance, the Prometheus operator knows how to manage Prometheus. Specifically, it knows how to take a bunch of rules from different namespaces about how to monitor applications, bundle them up into something that actually can be passed to Prometheus itself as configuration and manage some number of Prometheus instances. So all operators are controllers, but not all controllers are operators.

A

Some controllers just have general knowledge, so the deployment control art doesn't have any specific insight into what it's managing all it knows is that it needs to do some rolling updates, all right so generally, over the course of this presentation. I'm going to talk to you about some concepts, then we're gonna, look at some practical details and then I'm gonna pop over and try it out and y'all can follow along and try it out as well.

A

So we'll talk about scaffolding and design and then we'll move into types and then we'll move into the code that makes those types do something useful and then, finally, at the end, we'll talk about moving from your test dev loop into something that can be deployed on an actual cluster for production use all right, so without further ado, perhaps a little to do.

A

If some of you think you have seen similar things before it's probably true I've, given some similar presentations before, but in this one, we're gonna talk a lot about the new features in kubernetes that make this even nicer. Some of the stuff that just came out recently right so cube. Builder cube builder is a set of opinions and building blocks, so there's kind of three components: there's queue builder itself, which is scaffolding out new projects and then there's that's built on top of two libraries.

A

The first is controller runtime, which is like a standard library for writing controllers and the second is controller tools, which is a set of tools to do stuff like compile your go types into a custom resource definition that you can install in the cluster, so that you don't have to write open, API SPAC by hand, because that is not a fun experience. In my opinion, alright. So today we're going to be building an operator for the guestbook tutorial from kubernetes, so who here has played around with the guestbook tutorial before at all?

A

Alright, so cool? Some of you, it's a pretty simple application. There's a small front-end, PHP application that connects to a Redis instance and then just basically lets you write in a name stores, the name in Redis and when you load the page and loads, all the names from Redis, pretty simple. But it has a few moving parts and so it'll be good to explore kind of some stuff on how to write an operator so we're gonna manage and deploy both and then we're going to expose the whole application in terms of a service alright.

A

So there is a repo for the finished code for this tutorial, since we have an hour and a half I'd recommend y'all take a look at that that might help you save you some time typing. It is at present dev, slash cubic on u.s. 2019, slash code and I am going to give people a few minutes now to pull that up. Also, if you put chopped off the slash code, you can follow along with the slides, if you ever want to take a closer look at something.

A

So this is all just on my website once you're in that repo there's a directory called goals. That is like the Amol from the tutorial. So if you want to look at like the stuff that we're going to generate on the cluster, you can take a look in there.

A

So give like 30 seconds for people to copy down that URL.

A

All right, so I sent out homework ahead of time. Hopefully at least some of you who've done the homework. But if you haven't you're, going to need a copy of Q builder you're going to want a copy of the complete repo, probably- and you might want to pull open the original tutorial in your web browser just to take a look, and it might help to compare and contrast what we're doing in the operator. So.

A

Oh yeah, you can totally you can brew, install if brew has to 2.0, you can totally install from brew yeah. We have a couple of contributors that maintain a brew repository at this point. You're also, if you don't have a cluster up and running you're, also going to want to kick off launching a cluster, so I'm just going to quickly flip over to my terminal. For those of you don't have one and I'm just going to show you what I did so.

A

I already have one up and running for the sake of this tutorial, but so we can see. I have created this VM here it is a container optimized OS image because we're just going to be running a cluster on it. For the most part, if you can't get other stuff running on your system, you might want to do. Is the Debian image instead, so that you can install other tooling but I'm, just using it to run a kind cluster.

A

So I will just briefly show you what I did is we see if we test the internet right now, cool right? So once.

A

Largest there we go awesome so for those of you who don't have a cluster I'd recommend using kind. So the first thing you're going to want to do is download kind, which is here. If you go to the kind web page, so kind dot cigs gates, do you can get this very long URL? So don't try to copy it from here. Just know that you or if you, google, kubernetes times you will find the right web page and then, if you're on container optimized OS, you just need to move kind to VAR run and then.

A

Make it executable like that and then finally, you can do VAR run kinds, create cluster and kick that off. It'll take a couple minutes: I'm gonna pop back to the presentation, but it's good to get. It started now: okay, cool on with the show. So how do we get started with queue builder? So what we're gonna do is we're gonna make a new directory, since this is a go project, we're going to initialize the go module in it and then we're going to scaffold out the basic queue builder code.

A

This will create some initial project structure, creating directories for our API types and for our controller and give us some nice deployment files for later on. We give it this domain here. This is just the suffix that is put on the end of our API groups, so that we don't clash with anyone else. Alright, so I'm gonna do that now up over here, C will create a new in progress.

A

You all see that let me make it bigger.

A

Go mod and knit coupon you can name this go module whatever you want and then we'll do queue builder in it. Jemaine I'm gonna use my personal domain.

A

Please don't use this domain. This one's mine, you can choose whatever you want. If you have your own domain name use that you could use example.com if you really want to it's reserved for testing so right and it's just going to scaffold out a project with the default dependencies it'll hint at our next step. If you want spoilers, you don't want spoilers, ignore that last line, okay, cool, so ugly, alright! So I'm! You heard me mention API groups a moment ago. So let's talk about how we describe kubernetes api sand.

A

Will howl explain what API groups are so in kubernetes we collect API types together, and so we call each collection of related API types in API group API groups can have multiple versions. This is how we make breaking changes to api's in kubernetes. We add new versions and then we convert between them and within that API group we call each type a kind. Now. This is most of what you need to know to talk about kubernetes api terminology, but there is one more thing that may come up. If you get really deep in.

A

You may also hear people talk about resources, it resources a use of a particular kind than the API, usually they're one-to-one, but not always there's a few exceptions in certain parts of the kubernetes api. So just you can think of resources as functions and kinds as return types, but usually in kubernetes. We only have like one set of functions that returns a given return type, and so when we have go types usually each of those go types corresponds to a single group version kind, all right.

A

So the structure of every almost every kubernetes kind looks roughly like this, so all of them start out with some metadata. This provides information about what kind we're dealing with what API version in this case we're looking at the pod in the core v1 group. We for historical reasons. We don't write the core, but it's kind of implied, and then there's also some what we call object metadata. So this is things like name namespace labels.

A

Hopefully y'all are familiar with that and then almost all other types almost have spec and status, so the spec contains the desired state and the status contains the observes date right. So practically speaking, the way we get started here is we call queue, build or create API. We pass a group name and then I'll get our domain appended. So if I say group web app, it'll be web, app doesn't matter magic hole, depth, we give it a kind name so we'll say guestbook here and then we give it a version.

A

We're gonna be bold and we're gonna decide that we're going right to stable protip. Don't do this and that'll give us something that looks like this for our API types, so you can see up top here. We have our spec and then we have our status. And then we have this root object, which contains our metadata, so type meant an object meta. So that's like API version and name and namespace and stuff, and then it also contains our spec in our status. And finally, we have this list type as you might guess.

A

This is the type that's returned when we call west's. So if we do cube cuddle get without a specific name, it goes and says to the API server. Give me all of the pods and the API server returns, this kind of record type, all right. So, let's look through each of those sections. So first up is the root type. We don't usually edit the go code of the root type because it almost never changes between types, but what we do do is we can add additional metadata to the root type.

A

This describes more information about our CR D, so the common things that we do is we enable sub resources and we add print columns so we'll talk about the status sub resource a little bit later in the tutorial, but for now just know that we enable it here with this marker comment. So that's that comment that begins with a plus and then we also had print columns print columns are the way that we can get nice cube.

A

Cut' will get output, so you know like with core types: if I were to do cube, cut' will get pods. I would see like some nice columns, that would show like node name and maybe pot, IP and stuff like that status, whether or not it's running or terminated or in progress. The way we do that for CR DS is we had these print columns, so here we've added to print columns.

A

We add one for URL which we're gonna be exposing in our status and we had one that shows the current desired replicas from our spec, and this way, when we type cube cut' we'll get guest bucks instead of just seeing name and like creation timestamp, we see like some useful information. That is a summary of all of our guestbook instances right on to the spec, so spec describes desired state like we said before. So we need to decide what we want to put in here.

A

So I've put in two kind of fields in the root of the spec. One of them is the name of the Redis object that we're going to use to fetch our connection details so we'll also have a Redis kind, and we will use the Redis status to know how to connect our deployment to Redis, and then we also have details about how to deploy and manage our front-end, so I've added resource requirements. So we can control how much CPU it uses and also information about the service and the number of replicas, and you can see.

A

We also can use markers here. Marker comments, this time they go in fields and they control stuff like defaulting and validation. So we actually have a pretty powerful language and open API, and all of that is exposed to these markers.

A

So we can say: don't let people submit ports that are negative, because that doesn't make any sense if they don't add a port, make it 8080, because that's a same default for instance, and we can say that certain fields are optional, so, like port is optional will default it all right status is very similar to spec. There is one thing to note here as you're thinking about your api's, which is to say, status, is used to convey information to the user and to other controllers.

A

It is not for storing things, so you should think about status as kind of ephemeral right like imagine that, at any time someone could come through and delete your entire status. You need to be able to recreate it from everything else. So, a given controller, that's writing status to its own object should not be reading in that status. It should not be depending on information there.

A

It's it's for it to communicate information out to the world right, so we're also going to need Redis types kind of similar to what we talked about for guestbook Redis has some one leader and some number of follower instances. So we let you control the follower replicas a couple things to note here. We can use Joe doc to set basically API descriptions that come out in our open API.

A

This is nice if people are generate using nice UI is that or IDs that give them help when they're writing their mo, and it also shows up in cue cuddle explained. So if someone says cute cuddly explained, guest books, it'll read from the open, API and I'll show like follower. Replicas is the number of follower instances to run so right.

A

So if you were paying super close attention, you might have noticed that when I default, 'add two fields in the guestbook I use slightly different types for them, so one of them was a pointer and one of them was not a pointer. This is a little bit of an important point in kubernetes, so if we don't have a pointer, the zero value is assumed to mean unset.

A

So in the case of integers, for instance, that means zero is the same thing as I didn't fill in a value, and so the default will always be applied, even if the user passes zero. So this is great for things like port, because zero is kind of a nonsensical port, but it's not so great for replicas, because it means we can never stop. Our thing right, so the solution to this is to make the zero value something else and the way we do that is by using a pointer.

A

So if the zero zero value of a pointer is nil, which means unset becomes nil, becomes a default, but if it's explicitly set to zero in the mo, that becomes a pointer to zero and we don't do our defaulting. So we let our users scale down our things, which is good because sometimes they won't want it running.

A

We can use almost any type in go in our spec and status. With a few exceptions we don't use floats because for those of you not familiar with how floats interact and they're they're very good at what they do, but that what they do is probably not what you expect, so they don't round-trip very well through through Jason and llamó and through other languages. So we use a different representation instead called resource stock quantity. You've probably seen it when you're sending your pod resource requirements. We also don't really use interfaces in our api's.

A

We prefer these kind of tagged unions down here where we have an explicit type field and then multiple kind of variant fields that are generally pointers to things or things with zero of all values.

A

All right. So let's try this out. So we're gonna populate our scaffold. It out sample so by default queue builder scaffolds out kind of an empty sample with some bogus data in it. So we can fill that out, I'm going to add some bad data. That's an invalid field into mine just so that we can see that validation is working and then we will actually pop over and generate our CRD from our go types with make manifests. We will install our CR D with cube, cuddle, create and we will attempt to install our sample.

A

It will fail and then we will install it again and it will succeed because we will remove the bad data and we'll be B. We will be able to run cube, cuddle, get guests books and it will work. So let's try that out. Just gonna flip back over to over here. So first I'm gonna fill out I'm gonna. Do all that stuff we talked about at the beginning, so we'll do Q builder create API, we're gonna, get a API and it's gonna ask us: do we want to create a resource? We're gonna say?

A

Yes, that's do we want to create those go types and do we want to create a controller? We'll also say yes, we'll deal with that in a little, so it's going to scaffold out some types, that's going to attempt to run, go imports to clean it up a little bit and then we've got this. The scaffold out types.

A

So I am going to cheat a little bit because I'm standing up here and I'm gonna copy over from the completed tutorial, so API v1, guestbook types and then we'll also look at you know: API v1, guestbook types in this, not split so you all can see both sides. We can see. We've got this empty here. We've got this complete here, so I'm just going to grab my spec fields from here and copy them over get rid of this nonsense and then you can see we have like this nested front end spec.

A

So we'll also need that and then finally, our status with just the URL, we could put other information in here too, but oftentimes. We have stuff like status conditions that convey kind of what the current state is of the deployment of the thing so like did, we create our service successfully, did create our deployment successfully and then finally, I'll also copy in all of those markers that go on our route type over here.

A

And then, finally, last, but certainly not least, since we use that core v1 dot resource requirements, that's the same kind of structure used in the pod container resource requirements, we'll need to import core v1 and then my compiler is happy, so I'm all set I can run. Make manifests.

A

Oh one more thing, sorry about this, so we're using brand new features, and so we've got a tell controller gen that we want to generate in the new format with those new features. So this was just launched in kubernetes like one version ago and so by default we don't generate that so that people get projects that work on older versions of new kubernetes. But here we like living on the edge, so we're gonna say use the GA version.

A

So we open up the make file and say CRT versions equals v1 as opposed to v1 beta 1, and let's try that again cool alright. So let's take a look at that quickly. Just so you can see that all the stuff you didn't have to write so we'll do config CRD basis, and you can see we get this nice open, API spec. That has all this validation in here we get some defaulting. You can see.

A

We had that faulting on replicas, with the default of 1 in the minimum of 0, and you can see that we have all this nice resource requirements stuff in here too. So, let's also edit our sample.

A

So it's under the config samples, guestbook and we'll give it a Redis name since that was required by this sample, will generate the Redis one later later and then we'll give it some front-end configuration so I'll, say, resources, requirements or requests, sorry, say: CPU, 50 ml. Of course it's not a big application and then we'll also say serving port whole and negative one, and that's just invalid right. You can't have a porch of negative one, and so our validation will catch that all right. So at this point all of you should have running kind cluster.

A

If you were on GCP, it should be up and running, and if you were not on GC p, your local cluster should hopefully be running or your remote cluster or whatever, whatever you're deploying on to so what I did because I'm running against GC p is that I as saged into the node, the compute instance I grabbed the cube config that was generated by kind, so kind, we'll say it's.

A

Your cube, config is now set it's at Tilley, slash, dot, cube, slash config, so I copied that to my local machine and then I'm going to run this fancy port forward command and I will show you how to figure that out in a moment, so swell commands over here. I'm, just gonna grab this thing here.

A

So basically, this is just the default g-cloud ssh command that you will get if you press the little drop down next to the instance box and say next ssh and you say ssh with g-cloud and then we're gonna pass this dash out here and f + n, which is going to port forward from my local machine to the remote machine, and so we're just going to use the connection string.

A

We got from the cube config, so I'll show you how to get that in a moment, but we're just gonna run this here, F and and just say: no I don't actually run it and want to run a command just go into the background in port forward and with any luck we should be able to do cool. So we have a working cluster right. So let me just show you.

A

Console clouds at google.com and we'll go under our compute engine instances here and then so the way I got that crazy command was I. Just went down to this little drop down here and I said you g-cloud command and you just copy this whole thing and then at the end of it you just add, and then L and the port and then the connection string from the cupid favour.

A

Please, if anybody is having issues with that or is confused, please raise your hand and we can have someone come over all right so now that you hopefully have a running cluster. Let's install the CRD.

A

Right and we'll also attempt to create our sample and we're gonna get this nice message here see so our validation is working. It said invalid value.

A

Spec front end that serving porch should be greater than or equal to zero, so we're gonna go into config samples and get rid of serving porch here, and then we were going to try again. It works because we provided value valid data and then, if we do cube, cut' will get guestbooks.

A

We can see. We have this nice table display. That shows us useful information. Just because we put those markers in place. We don't have any URL, because we don't have any controller to set the URL. So let's tackle that I'm gonna give everybody a couple more minutes. First, though, to try this out and remember, all of this stuff is in the completed, get repo.

A

So if you are having trouble or you think you type out something- you can't figure out what it is or if you just don't feel like typing I, frequently, don't feel like typing. That's why I have Kay Elias to cue cuddle instead of having to type out cube cut' all the time you can just go from there and if you really don't want spoilers in that repo I've got the exact git commit where you should be checked out to in the completed repo.

A

If you really want to just follow along exactly, instead of just using the end right and feel free to raise your hand if you are having issues.

A

And or if you think, there's something that's very confusing, you can also ask the question out loud and I will try to answer it on stage because it may be confusing for other people as well.

A

Would you say slow down a little bit, yeah I can I'll give you I'll, give you all a few more minutes.

A

And as a reminder, if you would like to look back at any of the other slides, if you go to present meta magical dev, which I can show you now.

A

You will get this webpage if the conference Wi-Fi is working and you can just click on this link here that says 0 operator a little zoom in a little bit. This link here that says 0 operator and you'll actually just get a copy of the slides. So if you would like to look at the slides at all or look back a few slides or whatever you can follow along there as well, can also look forward a few slides, but that would be spoilers again and that's no fun.

A

If you and if you want to ask a question, you can also come up here and I'll repeat the question out loud. If, if it's like, you think it's relevant for everyone.

A

Actually I'll come down and help somebody dull to just expedite this.

A

A

It could make my own.

A

Yeah, so I've got a couple of good questions. The first one is: what was that make file option and it's CRD versions, plural equals v1, and you can see that in the completed repo too and the other one is. If you get this weird error about, go dependencies that occasionally happens due to the way go likes to resolve dependencies.

A

If you go into your Godot mod file, you should see if you're encountering this issue, you should see two lines, one that says: controller runtime and one that says: kate's io, / client go. If you delete the line that says, kto / client go and then you rerun like go, build or go mod tidy or something like that. Basically, any of the commands that effect go modules.

A

It will attempt to reira solve dependencies and actually use the right version. Basically, kubernetes go dependencies. Management is like half way to go modules, and so occasionally it gets a little bit weird. Oh.

A

That changes in Godot mod, so it's your Godot mod, is your dependency less right. It's like your version of your all your modules that you're using.

A

The question was about the new features we're using in our CR D, and what? What versions will this work with and- and why are, why are we turning on this feature, so this is actually about things like defaulting, so those are only turned on in the GA version of.

A

Cr DS and generally you could hypothetically write a controller that assume the defaulting was not going to take place and like back up defaulted but use the CR d defaulting if it was available and just generate two different versions of the CR D. That's actually why the option is plural. You can actually specify multiple versions there and install a different version depending on what cluster it's running on that's kind of a little bit more advanced than this tutorial. I. Think so we're just gonna stick with stuff for the newer version.

A

When you do queue, build or create API it'll ask you two questions say yes to both of them.

A

Yeah, so actually, if you had that failure with the dependencies, it's just failure attempting to build I, believe you should have all the files still there.

A

Okay. So if you then deleted them, you can go into your project file. It's just called project and all upper case and there's like a list of types and if you just delete it, you should be able to run the command again.

A

Just a reminder, so you will need a 1.16 or newer technically if you're, compiling from head a cluster. So that's why we're using kind today, for instance, anything older you'll get messages about the v1 CRD, not existing, or if you try to generate v1 beta one it'll yell at you and say: you're not allowed to use defaults in v1 beta one.

A

Okay, so make sure if you're missing, config CRD bases make sure to run this first command, make manifests that will generate your CR DS and put them in that bassist directory. That's probably it alright quick show of hands who has managed to successfully run Q cuddle, get guest books. This last step, all right who has like gotten mostly the way there and like hit some speed bump okay and who is still waiting for their cluster to finish, creating because of Internet issues.

A

Okay, so, for the sake of time, just to make sure we get through everything I'm going to advance along a little bit and then, if we have time at the end or the next break, I'll be happy to come around again and and help out and Walter can also come around and help you out alright. So we have our types- and this is awesome, but at this point, kubernetes is effectively acting, acting as a structured, key value star, which is like great, but we probably actually want to do stuff right, deploy pods, run workloads.

A

All that jazz, see URLs have things show up in our browser? That's the fun demo endpoint right, alright! So let's talk about how we make things go. So when we talk about kubernetes- and we talked about controllers, the core concept that we talked about is reconciliation, reconciling our observed and desired states, and so generally in kubernetes. Reconciliation follows like a standard pattern. This is followed by almost everything in kubernetes, be it custom operators, custom controllers, built-in controllers, even like the qubit. So we read our route object.

A

The main object we care about for any given controller. We fetch all the other objects we care about. So these are might be objects. We created objects, we're reading information from we ensure the objects we created are in the right State. We insure that our external state, things like load, balancers databases, etc, are in the right seat, and then we write back what we did in to our object status and then we repeat over and over again. So let's take a look at what that looks like in code form.

A

So this is a little small but we'll go over it in more detail and I can zoom in a little bit. But this is in the essence, the entire reconciliation loop for our guestbook controller. So we can see we start out. We get this reconciliation request here and that contains the name in the name space for a guestbook.

A

So what we're gonna do is we're gonna attempt to load that guestbook and then once we've loaded that guestbook we're also going to attempt to load the Redis instance that it references so we're gonna, look at spec dot, Redis name and we're gonna construct a key, basically a name in the namespace and we're gonna say: load the Redis name, spectra Tasneem in the namespace, that's the same as our guestbook and then using information from that.

A

Redis is status and our guestbook spec we're going to construct the deployment to run our guestbook front-end, and then we're also going to use the information in our guestbook spec to construct the service that exposes that deployment to the world. And then, finally, we are going to submit those changes to the API server and update our own status and then update our the value of our status on the API server. And finally, we return. This result, value that result, value says things completed successfully and we don't need to riku.

A

So if we needed to do periodic updates or if we need to check again later because say we're reconciling some external status that doesn't implement this kind of watch mechanism that we have in kubernetes, we could say like riku after 10 seconds, to check again in 10 seconds. Well, we don't need to do that. Also if we return an error here as the second value, we would get automatically recued with back off.

A

So if we fail to do something, we can just return an error and our mechanics will try again later but backing off so that we don't just keep trying really quickly forever all right, so you might have noticed there were a couple helpers there I kind of moved them out, so that it would be easier to go over the structure of the code, but the helpers are actually pretty simple, so normally in kubernetes, we need to be make sure to be item potent with our changes all right.

A

So what this means is, if our object is in the right state already, we don't want to accidentally reset any fields that someone else set or change things. We just want to make sure that everything is in the right state that we care about, and so previously in older versions of kubernetes. This was kind of hard if any of you were have seen any of my previous presentations. You'll know that there's like a couple pages of code just to do this.

A

This has gotten a lot nicer with a feature called server-side apply which I like to call the Picard patch. So what server-side apply lets you do? Is it lets?

A

You declare the object in UML or Jason form, and you just set the fields that you care about, and you always set all the fields that you care about, and then you tell the API server make it so, and the API server figures out what changes if any need to be made to the actual object, and this allows multiple writers to make their changes and just say ensure it's the way, I want it without stomping over each other.

A

So here, even if say, the service controller, populates some more information or add some defaulting in, even though we haven't specified that defaulting we're just constructing it from scratch. Every time we won't clobber out those changes, and so you can kind of see this go code almost just looks like we've written, kubernetes Hamill and go, and that's basically because we have it's very similar to how cube cut' all apply, works but way more powerful, because multiple writers can do it and it's on the server. So we don't need any special logic, client-side view.

A

One other thing that we're doing here is we're setting an owner reference. This does two things. The first is that when we delete our root, guestbook object it'll, make sure that our service and deployment are also cleaned up. This is called garbage collection in kubernetes. If you want to google for more information about it. The other thing it does is a little bit later on when we go to wire up our reconciler, so that it's actually called it. Let's, our controller machinery know when you see changes for this deployment.

A

It's actually we need to be reconciled. Instance, the other one here is pretty similar. It's how we construct our status information. You can just see, we load the service, we set some fields, we always set the value. So if we don't have any information, we explicitly say that, and if we do, we say that okay, so now we just need some wiring. We need to tell it when to run. We have our logic in place, but it's not gonna do anything.

A

So when we talk about wiring kubernetes controllers, we talk about relationships between objects, so each controller is for a single kind. It can have other kinds that it cares about, but there's this kind of route kind that the reconciler functions in terms of, and so our case for a guestbook controller. The route kind, as you may have guessed from the name- is guestbook, and then we talked about two types of relationships. One is a very common relationship and that's the owns relationship.

A

This is for objects, we create, so we define that relationship in terms of that owner reference that we just set and then there's another relationship which is more free form called watches. This is for objects, we didn't create, we probably don't modify, but we do care about some values in them. So, in the case of our guestbook, if we flip back a couple slides, we can see that we pull information. Specifically we pull up, we actually don't have we'll see. In a moment. We pull information for constructing our deployment.

A

That was just the service from the Redis status. So we care when the Reta status changes, because if it changes service names, we need to know so we can update our deployment configuration. So we say that guestbook watches Redis alright. So what does this look like in terms of code practically?

A

So you can see. Basically, our reconcile. Our logic goes in this method called reconcile that implements the reconciler interface. Basically, what we saw before is everything in the reconciler. The only difference is we set up some context and logging information just so that we can see what's going on, and then we put all of that log code that we just wrote in the middle here should look fairly familiar all right and then we need to register that reconciler with a controller that knows how to trigger it.

A

So we say we want to create a new controller managed by a manager. The manager is just responsible for setting up, watches and all the boilerplate and running our controllers and waiting for signal handlers. All that, like boring stuff, that actually just we don't really care about what we need to do for everything. And then we say that this controller is for guestbooks. It owns services and deployments just like we talked about, and it watches Redis instances now watches, as you can see, is a bit more involved because it's an arbitrary relationship.

A

We need to specify what that relationship is, and we do that using a mapping function.

A

Now the mapping function needs to be able to map from a Redis instance back up to a guestbook, and so to do that we need to be able to quickly look up what guestbooks reference particular reticent sense, and so we do that by setting in index you can see this index is pretty simple.

A

Basically we just say what is the Redis name, and then we index on the Redis name well see we'll use that in a moment in our mapping function, so the mapping function is also pretty straightforward. Basically, the meat is this function right here. So all we're doing is we're saying, given some particular Redis instance list all of the guest books but filter on that index.

A

We just set up so that we only get the guest books that match that have spec dot, Redis name as whatever the thing we're reconciling is and then once we get all the guest books, we just convert them into these reconcile requests, and all that is is just a name in the namespace. So all we do is we go through each of the guest books. We pull out the name of the namespace and we put it into this.

A

Reconcile request, object and we return it and then our controller pushes all those onto the queue to get reconciled all right. So that was a lot, so we're gonna actually try that out now. What I'd recommend is that you copy and paste that from the completed repo, you totally can take the contents in the presentation and copy them in or write it out by hand.

A

But I'm, assuming most of you like me, are a little bit lazy and wouldn't want to type all of that deployment go object out, so I would recommend copying and pasting it and then, when once we've done, that we can do, we can run make run and that will run the controller locally against our remote cluster.

A

This is super useful for dev test cycles, because we don't have to bundle it into a docker image and then deploy it on the cluster, wait for it to deploy and we can even do stuff like run it under Dell to debug. If something goes terribly wrong, so we're just going to try that out. So I am going to go and copy over the code myself and then I'll come down and answer questions from y'all.

A

Yes, sorry about that.

A

Remember what the directory was called.

A

Okay, cool all right, let me make that bigger cool, so I'm just going to go into controllers here, and we see we have this guestbook controller and it kind of looks like what we just saw. We have this empty reconciler, so I'm gonna split with the completed one and we're gonna grab the complete controller, which is effectively just what we went over before all right. I ignore some of this extra stuff.

A

On top we'll talk about that in a bit I'm just going to copy go over of the internals of this scaffolding here copy all of this stuff, which is just what we saw exactly and I'll paste it back in here and.

A

Then same thing with our setup here and I've actually moved that help those helpers that we saw out into their own file just for organization purposes, so I'm going to copy that over we're also going to need all of the core API types that we referenced. So let's put those in place right.

A

So basically, these two are the kubernetes api types for deployment and service, and these three are the bits that we need for writing. That watches relationship. All this says go doc. So if you're interested you can go and look at the go doc as well. It's at Godot org, so I SIG's, a kate's that I have slash controller runtime, there's a link at the end of the slides too, all right.

A

So this is currently going to complain if I had to guess that we don't have any of these helper methods and we don't have a Redis instance so I'm going to really quickly create the reticence since I'm gonna copy that over I forgot to do that earlier beats.

A

A

Version the one group web app just like we did know just like we did for our guestbook, it's gonna complain, then I'm gonna copy over all the stuff that we're missing.

A

And then we'll copy over that helpers file, which is exactly the code that we just talked about.

A

Under controllers cool and so now, if any luck.

A

Hey Rach, look. We should be able to do make run.

A

Let's regenerate for.

A

Yeah, that's why sorry copied some stuff incorrectly? Let's fix that fix that up perils of live coding right.

A

So we called this Q con instead of workshop code and the this version, so we just need to make this cube con right.

A

Let's try that again cool, so we'll install our Redis CRD as well, since we have it. I haven't done that yet I don't know if any of you have but I haven't done it well make sure cube. Cuddle is on my path and then we'll do that cool. Now we should be able to connect, run.

A

Aha, we can see, we've got this log line that says successfully reconciled our guestbook controller, so we have a running controller for guest bucks a little bit later, there's also code in the repo for the Redis controller. I did not cover it in the presentation, because it looks almost exactly the same as the guestbook controller. Instead of creating one deployment in service, we create two deployments and services.

A

That's basically it alright. So let's take a look at the time. Cool I'm gonna. Give you all some time to try this out. If you are confused or unclear about anything or something's, not working as always raise your hands.

A

We just turn it.

A

Just to reiterate, if you're looking at the complete repository you'll also see those Redis types, I just mentioned, and the Redis controller so for the final full working thing, you'll also need the Redis controller. If you've got the guestbook controller up and running and you're curious, you can look at the Redis controller, but it should look very, very, very similar to the guestbook controller we walk through. We create two deployments, one for the leader, one for the followers.

A

We create two services, one for the leader, one for the followers and then we write down the leader and follower service names in our status.

A

But again, if you're getting that error with like wrong number of arguments and watch, it's basically due to pulling to new of her version of client go. So if you go into your Godot mod, you should see a number of lines that start with just cait's dot IO. It should be API machinery and client go and API. If you just delete those and then run make run, go we'll reevaluate the dependency graph and find the right versions.

A

This is actually a good point to bring up so modifying our controller. Does not change our CRD definition. So once you add, the Redis types in the only thing you need to reinstall hold on is the Redis types. The guestbook types haven't changed from when we generated them before. Even though we've changed the controller itself, the controller does not affect the CRD types.

A

And one other note: if you are running in kind, do system last-minute changes that URL is not going to show up, because it's waiting to see a load balance service, so we could and I will, if you want, if we have time, modify the service status code to instead of looking for a load, balancer, vez or waiting for a load. Balancer vez just do a normal service and put say the kubernetes api prefix in there instead. But it's basically the same premise:.

A

As a quick as a quick show of hands for a status check, how many of you have gotten make run running right? How many of you are still copying code?

A

How many of you have not gotten to this slides point yet and are still trying to work out stuff from the last pause? Okay and last but not least, how many of you are encountering issues with make run.

A

Cool all right, I will try to come around if you have issues with make run especially raise your hand male to try to come around and figure out what they are, and hopefully we can take care of some issues for the group.

A

So if you are running locally, you are docker pulling over a conference Wi-Fi on your cluster. This may take a while. Yes, you should see a deployment all right.

A

Okay, so if you copy code over from the completed workshop, make sure you change the import path so, like my import path for the example, workshop code was workshop code as the base, but if you did go mod in it, mycube con you're gonna have to replace workshop code with my cube con, for instance,.

A

For those of you who are not seeing a set of deployments as I, am we have a quick for this, and this is cuz? I was lazy and did not add sufficient logging to my controller. So this is a lesson to y'all make sure you have good instrumentation.

A

A

Let's, uh let's figure this out: this is a good good learning opportunity. So, if something's going wrong in our guestbook controllers, it's not creating our deployments I'm going off the script here. Let's hope it works. Alright, so something's gone wrong. So the first thing we probably want to do is we want to add some logging. So let's do this we'll say: got Redis now I'm cheating a little bit because I have a very strong idea of what's going wrong wrong and then we'll give a controller.

A

Runtime uses structured logging by default, so we can cast key value pairs after our message. So in this case we're gonna say, got Redis and then we'll give the Redis name so Redis name like that. Alright, so now we're gonna run make run again and we should see that we don't actually see got Redis. Alright,.

A

So I had to guess.

A

Or we got an empty rebus, so let's try that again.

A

I'm guessing that we're hitting an error there, yeah see there, we go, didn't guy, read us and that's cuz. We never created a reticence since whoops.

A

So let's edit our Redis sample.

A

There we go that was rather quick. All we did is set the follower replicas to one I. Think we set a default for that by default, so we didn't even need to do that, but we'll do it and then we're gonna do.

A

All right, let's try that again and in fact you should.

A

There we go so we see we got, this got reddest line and we can actually see this is kind of cool with our structured logging. We have these key value pairs here and we see the ret. We got the Redis ready seam. So let's try that again.

A

Tada, alright, we actually have something that works now.

A

Cool so I'm going to move on because we're running a little bit low on time and then it's a very end I'll be happy to come around and help anyone else, who's still having issues all right. So before we are done, we need to talk about how to package this up and run it on the cluster. So there's two things we need to talk about and the first of which is permissions so right now, I'm, guessing y'all are running the controller as cluster admin. This is not great.

A

Do not run all of your controllers as cluster admin. I can safely say this without being a security person.

A

So the way we control permissions in kubernetes is generally through something called our back role based access control, our back functions in terms of roles, as you might have guessed, and bindings of users to those roles. So by defaults we generate a role binding that binds the default service account. That's like a user for a controller in the namespace. We're running to this role that automatically gets created by queue builder queue builder, scaffolds out markers to generate rules for that robot.

A

For that role that let us read our own types but since we're also reading and writing other types, we need permissions to read those too. So we'll add these two markers in here. So this one says for the group apps and the resource deployments.

A

We're gonna allow lists watch, so that's so we can list and watch and trigger reconciliation. Yet so we can fetch and patch patches. The the underlying verb behind server side apply same thing with services. We need to get list, watch and package, so that'll generate these roles and then, when we run make manifest it will regenerate right. So the other thing that we probably want to talk about is running this properly. So.

A

Cube builder scaffolds out make file rules to build and run your controller as a container on the cluster as a deployment. In fact, so this first command here will actually build all your go code and then it will push it up to a registry. So for those of you who have GCR or GCP accounts, you can push to GCR if you'd like so you got GCR slashed your project, name, slash and then whatever you want and then we should be able to deploy and that'll just launch a deployment on the cluster with that image.

A

That runs our controller and if all went rel well and we're running both the reddest controller and the guestbooks controller, we should actually have a functioning guestbook instance.

A

Alright, so I'm going to switch back here and just for quickness is sake. I'm going to switch to the complete one because that's got everything all set and then I'm going to cheat a little bit. I don't actually want to do all these doc. Our operations in my laptop because it's gonna be slow, so I'm going to run them with Google cloud, build.

A

Which is like docker builds, but on GCP, so well, that's running I'm gonna come around and see who else has questions right as a show of hands who were able to who was able to run make run from the last step?

A

Who's still stuck at, like the make run portion alright and who's stuck before the make run portion. Okay, cool, hey, keep your hands up and I will come around to help out.

A

Under at this point, you can basically just go to the end of the completed repo so like where you originally cloned, or you can go to that sha at the bottom there, but basically we're at the end. At this point.

A

Anybody is all finished and wants to do some further reading or investigation. You can go to book, Q, builder, dot, IO and there's other tutorials there and whatnot. Just in case any of you have to scoot now or whatever, and want to see that before I advance to the next slide.

A

This were roughly out of time, I'm happy to continue staying around helping people out, but I figured I'd advance the last slide. There are some resources up here if y'all want to learn more and thank you all for coming out.