Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Hidden Generics in Kubernetes' API - Eirik Albrigtsen, Babylon Health

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Hidden Generics in Kubernetes' API - Eirik Albrigtsen, Babylon Health

A look into how apimachinery enforces strong conventions on kubernetes' api, how that impacts client-go, and how we can take advantage of these hidden generic properties to build an actually generic client in rust. We will go through a kubernetes Object, its generic parts as modelled via a rust Trait, and how this enables the rust compiler to automatically populate all kubernetes objects' api calls and serialization code from a single generic impl. We will also show examples on how to write single file, performant, async/await rust controllers that follow operator best practices.

https://sched.co/Zeit

A

Hello: everyone, my name, is irik and I'm one of the main maintainers on cubers and before we start here are a bunch of links that are relevant to this talk. If you're interested me sources slides, I go by clocks on github or this thing here on twitter.

A

But today I am primarily here to talk about the kubernetes api, some of the generic assumptions and invariants that kubernetes wants to maintain look at how these invariants are put in place, how they're enforced and how reliable they are.

A

We're also going to talk about how to model the kubernetes api in rust, using generics and see what that gives us in terms of consistency and, finally, we're going to talk about some abstractions about this rust api that ultimately lets you write lightweight controllers, so we're going to look at what kubernetes actually provides.

A

These are going to be our main sources and we're going to start by digging into the arguably most important file of all api machineries meta types, because this file really has everything that you associate with a kubernetes object like every object, has a kind and a version from type meta. That is flattened into the root object.

A

Every object has object meta. This is the core metadata everyone thinks about. This is a simplified view. I've hidden all the read-only properties, I've hidden, all the annotation, because everything is optional, but every object must have metadata, and it must look like this. So all these great things in here, like owner references labels, annotations finalizers, also manage fields that can go in there and if they are, then they are standardized.

A

There's also listable types lists for when you want a collection of items, this actually returns a different metadata which is not relevant right now and but, more importantly, there's a dynamic collection item there for items so that you can use put whatever inside of here at runtime.

A

Then we got all the stuff for api parameters, get options, list options as shown delete options, patch options, basically all the parameters that the api accept are encapsulated into constructs in this root file. You also got all the error responses for the api.

A

You got label selectors that sit inside the list option so there's a generic way of doing filtering and there's also api resource struct for standardizing all the meta information about a resource and how people can use it, and so that all of this and more in basically 300 lines of code, so I'm being very positive about this file. But it's really because of this file and the complete cons, the consistency and complete adoption of everything in this file that kubernetes feels so consistent, and it's also why we can make generic assumptions on other languages.

A

Now. Writing structs is one thing: how do we ensure that these structures are consistently reused across all the kubernetes? Let's look at client code, for contrast.

A

So here are the typed api methods in client code. This is just interface. You can see these old getters updaters patches, replacers, etc for deployment, and so all the actual implementation here ends up being about a 200 line file. You can go look at any other object like pod and you can see there's basically the same story. The only thing that really changes between here is the second argument: the object that we're acting on along with some sub resources at the bottom.

A

So there's a 200 line file for every object and there's a lot of object in kubernetes. So how could this possibly be consistent?

A

The answer is in the header like all of this is generated, and this might seem obvious to you, like you, you'd- have to enforce some of these assumptions for them to stick, but it's still kind of crazy. Like you got all this, you got all this glue to basically generate this code manually to you're, basically doing manual generics.

A

It is consistent sure for each kind, the specific structure are specialized via external code generation and the generated source has to be present somewhere, and it's just this repo, but you've got a lot of code that crucial code. That's in here like this, the client api is in here uh there's also. Not only is it a client api file for each type, there's also informers for every object.

A

This client setup for each group and as a result like client go, is more than a hundred thousand lines of code and that's without rendering- uh and I'm not here to really try and pass judgment at this. This is uh clunko is still great, like the fact that everything looks the same in here uh and that it takes advantage of api machinery consistently is what enables qcl to provide such a consistent interface, even if the language makes it hard for you to do so.

A

uh Amusingly there is this five minute clip from kubecon barcelona on the keynote uh that actually talks about this particular thing and also matches my motivations for writing. Some of the stuff that's coming up here later, uh but don't ever have time for that. So I'm gonna go on to the documented api concepts instead, so here, so there is meant to be a consistency between api endpoints and objects.

A

So this is how that would look. It's probably displayed on the apa concepts under kubernetes website, though here is where things start to break down a little bit, because this url scheme does not hold for pods nodes, namespaces services, pvcs, oh, where am I going or any other type in the core v1 list, because all of those types have a different url, presumably for legacy reasons, and that url starts with api rather than apis, and the group is missing.

A

So this is a relatively minor inconsistency because we can just special case this one empty group or core, but it is still kind of awkward.

A

The second concept. Api concept is watch events. So that's what you get when you perform a watch call on any list endpoint. This is how that looks. Basically, this is a two line response. I have split it for the presentation. You'll get a chunk response. Typically, one line per chunk, but you'll have to buffer yourself because each of the objects are embedded here can exceed the mtu and frequently does so.

A

But when you have a complete line, then you can parse the inner object as the type you actually want, and all apis use this concept, and it is consistent. The source for it looks like this and you'll find more of this runtime dynamic objects in there that handles different types.

A

So that's how kubernetes does some of these core things and I've actually covered all the concepts and main api consistencies that I want to be talking about a lot of it was the code generation.

A

A lot of it was done with dynamic types, so I'm going to be moving on to things from rus point of view now so like in the go code, I'm going to be simplifying slightly for readability, but most of the stuff here in is going to be from qbrs, but first we're going to start out with a project by arnov singh aka arnavian, because that project k8s open api is really the linchpin that makes any generics possible.

A

Not only does it generates the rust structures from open api schemas, it also factors out some of the consistency into a few traits that is then implemented for all these structures.

A

So a huge shout out to him like for what I believe is just his side project. I really cannot thank him enough, so his first trait is basically resource and the rust rate is that's the ldr of the behavior that you can implement for a type and then later you can use the trait as a constraint in function, signatures- and I say behavior, because you can't put dynamic data in there, but you are allowed to put static associated constants, which is what this is.

A

So we can use this to map an api to where, on the api it lives.

A

The second trait I want to cover is metadata, which really is just a getter or a generic way to grab metadata. That is consistent across all objects, because even if kubernetes always puts metadata on the same key, we can't really guarantee that. So we need some indirection and traits are the way for that.

A

This is how that would look. It's slightly simplified because kubernetes open api distinguishes between listable types and list meta that uses less matter, but everything else uses object matter and we from qbrs point of view, only really do stuff on top of objects that have object meta. So we're hiding a few details here.

A

uh But with these two root rates we can start and build a dynamic api, so you get kind of how it gonna work.

A

So this uh struct here it's got all the same information as the resource rate, but it also has got a dynamic, namespace property as well as all of these being strings. So you can instantiate this at runtime from an arbitrary object.

A

So while you can fill it in completely manually, there is this nice thing now about the existing open api struct. We can actually just provide a blanket constructor with one trade constraint uh that would look like this uh for a namespace resource, so we have all the data. So if this thing implements a trade, so we just copy all of that when constraining by it and as a result, the resource type is not generic.

A

uh We just use the trade constraints in the in the constructor, so with this information and this easy way to create resources, we can now start thinking about how to hit every object's api endpoint. So I'm going to make a very simplified url mapper, something that will let us hit basically every url endpoint in kubernetes.

A

uh So this is this is that uh it's gonna handle it's gonna match the api convention for for resources, and it's gonna handle that awkward. Empty group case that I talked about earlier. However, it is slightly awkward, we haven't, got all the information I really would like in the in the trade at the moment. So you can see the resource line actually got a load-bearing pluralization in there, uh because we only have the kind uh so yeah, that's not really a great phrase to use about software architecture, but uh let alone things that I'm designing.

A

But here we are ultimately uh with a url mapper in place. We can hit pretty much anything and this we can start making a dynamic api.

A

So we're going to focus on create for basic crud, so this is how crate would look. It takes a post parameter struct, which is analogous to the type in types that go. uh It also takes some binary data makes query parameters from post parameters and prepares a request. You must then execute the request yourself, so this should be pretty similar to how another language client would do it.

A

You get some bytes go through url mapper make an http call response, you get response bytes and you have to deal with serialization yourself, but that's not really what we want to be doing like we don't want to be manually, keeping track and mapping uh resources to serialization formats. We want what we really want is an automatic serialization detailization system, some kind of mechanism that is generic over k. That is aware of the underlying structure resource.

A

So for that uh we're gonna make our first truly generic type. The api- and this is a wrapper around resource with an http client handle inside of it, along with an empty marker of what type it's for, because we don't actually store any data 4k in here.

A

So this is just a marker for the type system we need to specify what type this api is for anyway, so it may as well be when we instantiate an api, uh so we can make an api namespace here by referencing resource namespace, I'm not going to show that, but this constructor would not take any anything ready to k.

A

So it's really when we construct. We have to tell the type system that it's really for pod here. So that's what this magic syntax on the left of the equality sign is it's a type hint.

A

So now that we have an api, uh we can't do anything with it, so, let's generalize create- and this is where all the funky rust and flags start coming in. This is what a generic implementation over a type k for a type that is also generic over k comes in. This is what that looks like here. K also is constrained by three traits it needs to implement and the methods create also further constraints k by an additional trait serialize.

A

So you can see that the body of a crate is basically just using the serialized rate.

A

Turning that data into json, uh passing it to a client that is done, executing the request and the result is deserialized into the type k because create actually uh takes an object at the same object as it will return so by using generics and constraints on k. We have actually implemented this client go like api method across all types with just a blanket implementation.

A

So that's really great. It's good success story, but unfortunately this is not going to help us for absolutely everything. I mean it's going to. Let us populate the api, but I'm gonna. I would be lying if I said we didn't need code generation, so uh yeah code generation does happen in rust, but uh contrast it is a required part of cargo build to execute and the system that does this is called procedural, macros or proc macros.

A

For short, I like to describe them as compile time decorators, because they're just these cute little things you put inside of in front of your structs or your functions or as ways to customize your so the interface of them end up being super compelling, although they can be slightly tricky to write the first time around.

A

But the benefit is really like the this because of the first class support for code generation. This whole class of errors, where you are operating on a stale version of generated code, is now eliminated. The compiler disallows that possibility you don't have to write a hapki make file with all of your stuff inside of it that glues this together, and so this is the example that pretty much everyone sees in was the first time uh surrey, fantastic, serialization library it.

A

If you slap these two things in front of a struct, then your struct will now implement serialize and deserialize, and you can convert it between various serialization formats.

A

You can customize it further with struct level or field level attributes in practice. You end up writing much of the same gunk. Slash annotations as you would with goes json encoding to like distinguish casings of your code and disk format, uh but you there's there's a lot of safety in in having this be top level.

A

Code like this actually goes through proper rosco to transform it into ros code and there's real good error, handling telling you where everything com came from, at least if the proc macro is written well, so we can actually make our own derivables as well, and so I'm going to be showcasing cube's custom resource brock macro.

A

Here we are telling uh rust to derive custom resource on fuseback and we're telling uh cube that uh the various attributes that this derived type is going to have like we're telling it the resource parameters, group version and kind, and it's then going to go and the proc macro is then going to create all the code around the custom resource that we're going to need and there's I've showed two lines of attributes here, but there's really tons more available.

A

We've actually tried to mimic a bunch of the cube builder setup for deriving custom resources, so there's a whole slew of things there for customizing how your crd looks, but what this actually ends up doing. It creates a food type. It attaches this spec that you have here onto the food type. It attaches the status type onto the food type. If you have one, I have specified one here. It attaches metadata type meta. It also makes a convenience constructor and a crd method to grab to generate a siri.

A

So if you wanted to start using foo now as a custom resource and query the kubernetes api against it, you really can just post the crd up to kubernetes with an api of custom resource definition type and grab the crd from foo in line two post it up, and then you can make another api of type foo and basically make new instances with your convenience constructor and start posting those.

A

Ideally, you would want to error, handle a new server side apply here. uh Maybe wait for the api server to understand this new type typically takes a few seconds, but once you've done that yeah, that's this is meant to be an illustrative example.

A

We've only really covered api creates here as well. So I'm going to dig a little deeper into some of the more advanced methods in particular watch because watch we'll need to talk about watch events first and that's the type that maps nicely onto the one in api machinery that contain the dynamic runtime object in rust.

A

Watch events can be packed into a fully generic num for a fully typed one, which is great. We also got some sort. The annotation here is to tell at sorry that the values in the enum variants go on the object key and the inum variant name go into a tag key. So this uh and the tag key is also sent under and received as uppercase, um that is, to match the convention and go for constants.

A

So you'd expect that a watch endpoint on api would return some form of watch events in the signature, and indeed it does a line further in the signature.

A

uh Unfortunately, it's perhaps more intimidating than at first, you would have thought you see the words impulse stream in there, which is a rust constraint that says that a return type must implement the stream trade, the stream trade being something that's, basically an asynchronous iterator, so you are able to iterate or over iterate over a stream, but you have to await each new element, uh the element being the item here, uh as in the watch event that can fail.

A

uh So you got a stream of watch event results, but on top of that, the stream is also wrapped in a result, because the http result can request can fail. So that has to succeed before you can even start streaming, and so that's why you got this fairly chunky type uh it. It kind of looks hard to use immediately and I've not even talked about the corner cases here and boy. Are there corner cases, uh so we gotta not only uh the document thing is like you, gotta track resource version. That's the first thing.

A

uh These are these integers that get passed on via cd, and you must pass these on, for every watch call to tell kubernetes where you left off. I kind of hid them from you, they're actually part of the signature of watch here, uh but uh I was sneaky and use the two character variable name, but it's there. It's required and the annoying thing about these resource versions. You've got to treat them as these opaque things and sometimes they're, just stale.

A

If you are sometimes you'll, just get 410 gone as a response back on a watch in kubernetes and if you're building a state cache like a reflector, then what you have to do is basically relist get the entire state back for every object in the system for that type.

A

If you get the synchronized and before watch bookmarks, this was super likely to happen, uh particularly on slow moving namespaces and partly as a consequence of this. If you are relying on removed events.

A

Well, that's now purely a best effort thing, because what happens if your app crashes, what happens if you get desynchronized in between an event being deleted, like that event, is lost if you've tried to initialize a watch from a zero resource version, which is what you do on the, uh which is what you can do, that at least you'll get an event for every existing item, but for deleted items. You'll never get that event again.

A

So that's not great watch calls also cannot stay open for more than five minutes, so you have to keep issuing this watch call for at least that frequency or you'll get more errors, and finally, just a share data use that you can get out of watch like on every change, including the status object if you've looked at nodes, for instance, like last, I checked, node status contained structs inside and for conditions, and each of the conditions would have a last updated timestamp and every few seconds this one of these conduct conditions would be updated, even though, like the boolean never changed, they would the timestamp would still be there.

A

So if you're subscribing to notes you'll get this whole hecking object every every few seconds, it's a huge amount of io, and you can. You can hide some of this from the user, of course, but this is still networked, so anyone who's writing a controller type solution on top of watch is gonna need some heavy abstractions.

A

uh So what would such an abstraction do? Well, uh we first got a list and then, from that point we gotta watch continuously, but no longer and for five minutes we gotta propagate all the user errors. We got a real list on desync errors, possibly losing events, keep track of resource versions and still somehow encapsulate it all in one live stream. So this is absolutely not trivial and that's why I'm so happy about cube runtime. So a huge thank you to mild maintainer tio.

A

He basically wrote this entire crate to figure out an entirely stream based solution for not only watchers but also reflectors and controllers, and because these objects are just this rust native concept of a stream, they end up being possible to manipulate in very standard ways like you can store them easily pass them around. Extend integrate instrument tests like we've, not even gotten, fully around to showcase nor proof of concept. All of this properly and so caveat.

A

This does have rough edges, uh but it's definitely the best evolution point so far for a controller on time and rust.

A

So I'm gonna be covering quickly how these three objects work uh and I'm gonna start with watchers and funnily enough, watchers actually end up being one of the more advanced of the three uh and that's entirely due to all the corner cases in watch that I spoke about internally.

A

We actually model it with a finite state machine and we are using something akin to a state transformer if you feel familiar with that term to pass around the state, which is the enum shown above and the actual watch events, which is the ultimate thing that we want to return.

A

So sometimes the watcher is actually going to be giving you a whole chunk of items which is what happens during your relist generally. You want to be working with a flattened version of the stream that watcher is outputting, so we have some helpers for that.

A

But this is the internal machinery. If you want to use it suppose you want to subscribe to added or modified events for config maps in some namespace, then this is how that would look line 4 here being the crucial one. You create a watcher on the configmap api and then you apply our helper to flatten and filter for only the applied events, and this is now going to handle all the watch complexity for you.

A

This loop should run forever and the fact that there's also an unflattened stream that we can work with means that any state store that we build on top of a watcher will always have some data even during a relist and that's why that's there? So let's talk about a state store, because that is what a reflector is meant to be like it's a it's something that builds on top of a watcher by recording objects in a store as events pass through.

A

So when watch events happen, we either insert replace or remove objects from a store, and then we pass these events unmodified just straight through. So it's a slightly complicated signature here for all this, you need a store which I'm not defined, but it's really just a hashmap of object. Reference to kubernetes objects, you're also going to need the unflattened stream from the watcher which is w, but ultimately this ends up just being this one line intercept on the watcher stream. So it's a really nice conceptual thing.

A

To use a reflector, you just make a writer and a watcher combine the two and then you can. You can use the reflector just as a watcher in the last three lines. In fact, those three lines are identical to the first example, uh but the more important part is the three lines in the center. So you not only do you have to create a writer, but you can also create a reader from this writer and this reader.

A

You can pass along to as state to a web framework like to expose it, because this thing can be cloned because, what's not cloneable is the writer, because it's kind of unsound to have multiple things writing into the same store. So this is illegal and by illegal and to clarify I don't mean it's a documentation convention I mean it's actually a compiler rust's move semantics makes the writer effectively disappear into the reflector and because it's not cloneable, there's nothing else. You have to do for this.

A

You, you can only have one writer and that's one of those great guarantees. You get that you get from this language that I'm super happy with.

A

So I'm going to be moving on to the final object that builds on top of these two, and that is the controller and a controller is a system that reconciles a root object or a custom resource, along with a child object that it may own or many child objects. And then it's going to call a reconcile function when anything related changes.

A

So the controller's job is just combining input streams from these various objects. Debouncing reconcile requests, scheduling, retries, and it's up to you as the application developer to make sure that the world is in the correct state uh when you build a reconciler. So when you get a reconcile request, you have to reconcile the root object and anything related to it that you own.

A

uh So there's a builder pattern to create a controller here, and that uh should remind you quite a bit of controller runtime. It's in fact heavily inspired by it and we even got some help from some of the maintainers on it.

A

uh So this here actually ends up being a completely sufficient main for running a controller.

A

This example tries to make a controller for a configmap generator, which is a custom resource that is meant to control a config map and ensure that the generator's state matches a configmap state, and I showed you earlier how to derive a custom resource from a struct earlier. So the only remaining bits here is how to provide an error, handling policy and a reconciler function that will be called with the context that you define so I'm going to dig into the reconciler here, because that's this is the signature of the reconciler. You have to define this.

A

It has to be an async function that returns a reconciler object if it succeeds and in the interest of not obscuring the slide. This function is where you would typically grab a client or an api from the context and start making api calls to kubernetes to ensure that the corresponding config map is up to date with the config map generator.

A

uh Maybe you also want to write something to the stator subject to indicate what happened or like how far along you got. Maybe it's a last modified timestamp, something that's not extremely noisy yeah, maybe I'm being a bit hand wave. Is this cheating like? How do you build controllers just build controllers, but this is.

A

We are designing your reconcile here, and this is really where all your business logic resides. I just want to emphasize that and all of the advice from controller building from controller runtime or queue builder generally applies here and there's tons of good talks on getting this stuff right in general, the tldrs that reconcile need to be item potent it needs to be able to resume and the places fail as in if parts of it that succeeded, then it should just keep going and follow on in the next iteration.

A

uh You should use server side apply to have lesser cases. You should use finalizers or owner reference to garbage collect because, as I said, you cannot rely on removed events reliably. So that's the general gist, but we do have some complete examples here.

A

There's two repos controller rs, which is a controller for a custom resource foo uh that lives inside of actics. It's got tracing, it's got custom, metrics, uh three files about 250 to 300 lines and then there's a version rs, which is a more lightweight deployment, reflector that exposes its reader through actics and that's a single line file 100 lines. uh So, ultimately we don't really want to be doing a bunch of scaffolding. Here, like what kabul you're saying, we want you to just choose your own dependencies.

A

Do you need a web framework? Yeah pick one, it's a pretty! It's pretty good practice to expose metrics. So at least a simple web server would be useful here. uh You probably want some observability tooling. uh Tracing ecosystem in particular is really solid. For this you can just slap on a hash instrument. Proc macro and add your favorite tracing subscriber sentry is great for error. Reporting prometheus great is great for custom metrics, so I can highly recommend all of these, but beyond that uh yeah you just have to investigate.

A

This is really all that I'm going to be talking about for cube, so I I've been talking for some time. I hope this has been helpful uh source for all. This is in the link. All of this is published on create style. Slides are available.

A

uh The api crate that I'm talking about has is has been reasonably stable, but cube. Runtime is still very new, so anyone who's willing to get their hands dirty help is appreciated uh if you are using it. Changes are documented in our change log, so please check that and pin your versions also. I want to thank bible and health who I work, for they do great things for kubernetes in the healthcare space and they are also great in their commitment to open source and have been particularly encouraging.

A

With respect to this talk so big, thank you to those guys as well, but that's really it so. um Thank you very much for listening to this and goodbye.