Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Polymorphic Reconcilers in Kubernetes - Advanced DuckTyping - Scott Nichols & Matt Moore, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Polymorphic Reconcilers in Kubernetes - Advanced DuckTyping - Scott Nichols & Matt Moore, Google

The explosion of Custom Resources in Kubernetes has lead to the development of new techniques to reference and reconcile objects in Kubernetes. Come learn how we are leveraging some simple patterns to produce complex systems within Kubernetes in the Knative project, and how you can adapt these methods to your applications.

https://sched.co/Uac8

A

Hello, this is polymorphic reconciler, z' advanced duck typing in kubernetes I'm Scott, Nichols, I, work on Canada venting focused on sources and I work at VMware, and this is Matt hi.

B

I'm Matt I, also work at VMware and I am focused on leading K native, serving a little bit of feedback all right. A little bit of background Before. We jump into the main event today, so custom resource definitions, aka CR, DS aka, the new hotness are leading to an interesting change in the kubernetes type system. We've gone from a world where we have a relatively small fixed number of types. If we want a reason about compute things like deployments, daemon sets jobs.

B

Staple sets types of compute to a world where any Schmo off the street, like us, can start to build higher-level, compute abstractions so, for instance, inque native serving we have this service and configuration resources. A bunch of different things in the the fast landscape, such as keema and open, fast and coulis have function, CR DS and this list goes on and on and on and on, and so we have this now new unbounded, ever-expanding space of compute abstractions. So the question we started to ask ourself is as controller writers.

B

How can we, if I, want a reason about not just one particular thing but the class of compute resources? How do I write controllers that can deal with this entire class of unbounded app resources so option a bake it it alright. So as these things come up, you just play whack-a-mole and you build and in and this gets really tedious.

B

So we started looking for an option. B.

B

Welcome to the untitled duck presentation.

B

This is our mascot view, lay.

B

Alright, so what we put on our wonderful hoodies, yeah Beulah, is going to be handing out the duct agon stickers in the back at the end. That's this wonderful logo. Folks inkay native I created an emoji this morning. Alright, so we gave a talk on duck typing in Barcelona. We'll put this link up at the end. You can see scott got to wear the costume.

B

Then we'll put this link up so that you can see it with a wall of links at the end, but we're gonna quickly recap what duck typing is and try and cover a largely new content and demos. So the if you saw that talk, you don't fall asleep. So without further ado, the untitled duck presentation well I'm, not keeping up okay. The first thing we're gonna do on our to-do list is quick recap, so duck typing so duck typing is here. We have three different objects right that have this common theme.

B

Each of these has a food bar property and, if I want to reason about this collection of different resources and reason about that food up our property duck typing allows us to define a partial schema that allows us to extract that information from different resources within this class.

B

All right, so how does this relate to kubernetes? So kubernetes actually has a number of instances where the idea of a partial schema is very relevant, so every single kubernetes resource embeds this type meta thing, this API version and kind. It also embeds object, meta. This metadata thing that everyone is every senior and gamal engineer in the room is very very aware of so the apps resources also define an interesting duck type, which is they all embed.

B

A pod spec in the exact same way, expect that template is this thing corvy one pod template spec, so they all embed the same shape and one of the things we showed in our previous talk is by leveraging this. We can actually take arbitrary resources that that do this, and we can extract that information. We can leverage patching to actually mutate those without knowing the full schema of these objects so cross that off our two doodles.

B

Next, we are going to talk about the concept of bindings, so what is binding so the way I think about bindings is largely in terms of runtime contracts. So, whenever I run an application, it runs in an environment that environment has certain facilities that may be provided by the system. An example of this would be the kubernetes service account token, an endpoint that allow you to sort of extract that information and talk to the API server, as if it were magic.

B

When you're running on a cluster, a lot of cloud vendors provide HTTP, metadata endpoints on the network, that you can talk to you to determine identity information and retrieve access tokens. There's also facilities like if I write to VAR log, it may what my logs may magically appear in some sort of durable logging system.

B

So as an application developer, this idea of binding is really powerful because it allows us to start extending that run, tract and interesting ways, so the application developer can just assume certain properties like those and build their apps with that assumption and that him that can be late bound into it by the environment.

B

So what are some things we might want to bind in right secrets, config, Maps sidecars, so one of the things we're going to show now is a proposed direction that builds on this, that we might want to use to implement a native event sources.

B

So we could call this POC, the sync binding. So there's a couple key elements of the sync binding there's the target: this is the kubernetes resource whose runtime contract we want to extend. In this case, we want to inject a simple environment variable into that binding target. That says: hey send events here, there's also the events sync.

B

This is the kubernetes resource that we want to send events and then the sync binding basically is responsible for extracting the information about how to send stuff to the event sync and projecting that into the binding target, easy right.

B

So the question is: how do I stick a sync environment variable into all of these, and how do we take all of the things we might want to deliver? Events to and extract the the address to send those things there? The value to put in the sink environment variable option a we baked it in right.

B

Okay, get the tedious option B as our duck types so for the target and the sync we're going to use two different duck types. The first is that class that we talked about a little bit earlier. Kubernetes application resources fit this mold and a few other resources like the k' made of resources do as well. Basically, if you embed a pods back at spec template, it forms a class we've started to refer to as pod spectable for the sink. This is OG duck typing.

B

This is what this is, what we call our addressable duck type and it's what the contract that has allowed us to effectively have surveying and venting be loosely coupled, but work really well together. So, oh so the question. The next question is: how do I write controllers that can effectively operate in this model?

B

First part is your traditional controller, so it reconciles sync binding resources and if I want to be able to have the sync change over time, it can basically extract that information, and can it's continuously reconciling where the sync says you should send events with a sync environment variable being present in the target. However, there's two interesting problems that just doing this is our traditional controller doesn't work for so one is immutable resources like a kubernetes job, kubernetes jobs, our pods peccable, but as soon as they're committed to Etsy D.

B

You can't patch it with where to send events. Another interesting case is when say, take a kubernetes deployment. We created a kubernetes deployment when we stamp out key native revisions, that's owned by another controller. If you start patching the deployment that we own we're gonna patch it back and we're gonna fight all day, and it won't end well so to address this.

B

We also run a mutating web hook. So this mutating web hook addresses both of these problems effectively for immutable resources allows us to perform the binding before the resource hit, set CD and for cases like a controlled deployment.

B

Basically, by doing this in the mutating web hook, it ends up looking like really fancy service like defaulting, and so well-written controllers already have to deal with the possibility that the create or update operation has more values filled in and deal with that. So that's the high-level overview of the binding, we're gonna walk through next, let's see a demo of it.

B

Okay, so I am going to open a whole bunch of examples.

B

Okay and the first thing we're going to do is I'm going to launch a simple container called debug and it has. It is an abundant aenor that I've just installed curl on and we're going to exact into it, and I just want to show you that if I echo, sync nothing right so our base, runtime contract doesn't have an environment variable named sink.

B

So now, if I exit the container, we have a simple sink binding defined here, where the deployment name debug, which is what we just created, is going to have the address of a service named sock I, basically projected into it. So if we, let's see, let's keep control this now.

B

Sakai is a tool written by this fine gentleman, I'll pull it up in just a second, because multitasking is hard all right. So, if I exact into this now, an echo sink will see. Hey the new pod has the sink environment variable defined and it has the address of where to send events. So the next thing we're gonna do is we're gonna pull up Sakai default, duct tagging that IO and we are going to grab this curl.

B

So as soon as we curl this. Basically the way Sakai works as it opens a web socket to my browser and as soon as.

B

As soon as I curl, this event joke's on you I'm a deployment shows up here so by curling the sink environment variable in that container, where we extend the runtime contract, we know where to send events, okay, so for the next example. This this is shows us doing it with a mutable resource like a deployment.

B

What about an immutable resource where, after it's created, we can't we can't change it. Alright, so I'm going to exit the container control example, no more bananas. As soon as this job runs right. We see hello from a job right and that the curl it ran. It ran the same container that is running in my deployment, except we set the entry point to curl it and send this event.

B

So now we've shown mutable thing immutable thing: let's, let's show what this looks like with a key native service which is a tree if sorry, a CR D that's defined, and we have no baked in knowledge of this in Sakai or the sync binding, or anything like that. Alright. So if, and here I have a simple proxy image all it does is it listens on EDA, T and every time it gets an event, it sends it to the what the sync environment variable says to send it to so. If I apply this.

B

And then we are going to curl a proxy and send it an event that says: I went through at a native service.

B

So, let's see as soon as I hit Send here boom I went through at a native service, so we've shown mutable. We've shown immutable. We've shown user-defined types so now we're gonna get kind of fancy, so fancy okay. So this is goes back.

B

One example to the job, except this version of the job, instead of sending it to Sakai directly, is gonna, send it to that proxy we just created, which is also a service, so also addressable, and this shows that with bindings really with the same job definition, the binding can actually completely change where that application is sending events. So if we apply example fancy as soon as we come over here, let's see hello from a job through a service.

B

Okay, so for the last demo, I'm gonna apply it first and you'll see why in a second, because this takes a sec, okay, okay! So now we're gonna get super fancy now and we're gonna. Do that same thing we just did except we're. Gonna have those jobs stamped out by a cron the cron job. Every minute is going to create a job that does the curl it's gonna label those jobs with app curl cron and in place of the direct reference.

B

We're now selecting those jobs with a selector, so every job, every job, that's labeled with app curl prom- is going to get sync bound in as its created it's gonna run, it's gonna send an event to Sakai or sorry to proxy, which is going to sentence a sucker. So you can see here that the cron has run curl a lot and if we switch over to here, we now have hello from a cron job via selector through ACK, a native service. Ok, so.

C

B

To recap: right visualize, it all right, so we created an immutable thing. We created an immutable thing. We created a user defined type thing that we use the immutable thing to send things through the the service. Then we created a cron job that stamped out jobs that sent things through the proxy onto the service so now I'm gonna hand things over to Scott, wrap up our to-do list.

A

Okay, so now we're gonna talk about dynamic type controllers, so Matt just showed an example of there's a reconciler, that's reconciling a concrete type in kubernetes. In that case it was a sync binding and inside it had two known duct types, that the controller could treat that as an interface, because the contract of the sync binding was that these two places the resource were pointing to adheres to this other kind that we we know about.

A

That's great, that's that's very useful, but what about the case where you have a reconciler that want that you would like to implement that reconciles on the duct type directly.

A

You can't do this in kubernetes, because all of the clients and all the caches and all the reconciler queues are strongly typed with a kind because you have to set the watch up. Kubernetes just doesn't let you ask for anything that matches things like this, so we have option a we could we could do manual config like it could think map. So we could.

A

Every time we install the new CRD, we could update the config nut and we could have some sort of watcher watching config map changes and stamping out copies of a reconciler injected with GPRS orgy, b, ches and it'll. The logic doesn't have to change, but how we load stuff changes a little bit. This works until someone deletes that CRD and then now we have a reconciler reconciling on an unknown type in kubernetes and your controller just starts throwing errors, and you I bet you that whoever deleted the CRD won't update your this config map.

A

So that's problem, but we have. We have more options. The thing that happened was that someone came to the cluster and deleted the the custom resource definition. So we could use that as a place to store metadata. So in K native we've been labeling our custom resources with the the duct type that they implement.

A

So, in this case, serving implements addressable like Matt was just showing there's also an object broker, it's also addressable, so we can label it that the duck type is addressable, and now we get to right at CRD, another reconciler that reconciles custom resource definitions, but this time it it filters based on that label that we've defined and then it can update that map internally, just in memory and stamp out. Reconciler is based on the types that it observes in the system, and this is awesome because if one goes away, we can clean up by yourself.

A

Yeah, ok, so now we're gonna demo. What a dynamic type controller could do, but first I'd give you a little bit of context: we're gonna leverage, Canada eventing, to do this, so we have there's a concept called a broker. It's basically an event mesh, that's the events go in. They come across the the broker and you can create a trigger that if the trigger matches the filters that you specified it takes the event a copy of the event and delivers it to a consumer with one little gotcha.

A

You can respond with a new event and it's just like it got ingress back into the broker, so another trigger could filter on that new event. So you can get kind of these cyclical loops, so we've already shown how we can solve the random deployment in two broker. But what about we have this problem where we want to make a function and we want to make a trigger, but if we make a copy of the function or the service or the whatever, you need to manage the the other pair the trigger as well.

A

So this becomes a little bit of a little cumbersome because you'll delete the service and you'll still have the trigger or you'll accidentally delete the trigger and the service is then a dangling node so started. Thinking about this and what we could apply this this duck typing concept.

A

We could have a controller reconciling all addressable types because we've we have that metadata and then I can stamp out copies of the reconciler that implements the auto trigger controller, and so, in this case it's a service and routes that reconciler sees a new service and it sees some special annotations and labels. It uses that metadata to create a trigger for the broker, and now we can automatically connect and we can. We don't have to think about that. Sorry, okay, so we're gonna demo this I mov, I user Matt uses Emacs, so that's gonna type.

A

So that's going to apply a cluster configuration very similar to what we just showed. There's gonna, be a random number generator, that's a sync bound into the broker and it's gonna make the random numbers every second between 1 and 100, so very similar to the demo that we showed earlier, and then there is point, there's a function that we call classifier. The logic is just it looks at the number and it says if it's odd it labels it odd if it's even it labels it.

A

Even so this is filtered on type and then the we're gonna send those new classified objects to just a copy of sock, sock I. But this time we're gonna use a cloud of an extension with the custom label that we our rear news. So it's a new event. It goes back into the broker and the new trigger picks it up. So if we go and open up an odd or even Sakai,.

A

And we start seeing odd and even events.

A

Okay back to the presentation.

A

Done, oh, hey, hey man. What is that there's a bonus task, Oh.

B

Visualize duck type relationships so.

A

We got to thinking and you don't just have to write, reconcile, there's that run in your control point for these duck types, the the duck types that we've been playing with help. You understand how the relationships between objects that we're creating so we're gonna show you a graph program. Well,.

B

That one on top looks familiar this guy's, so.

A

Above is the graphical representation of the demo previously, but graph. The graph program doesn't understand sync bindings at all. It only understands it can find objects that are installed labeled with that source, a source label. They can also see things that are addressable and we can infer connections based on this because we know the shape so there's that graph alright.

A

So you see that, like the normal one straight to Sakai and then the super fancy one that goes into the proxy and goes into Sakai, and then we see basically the the same set up we saw so we have a generator, goes into the ingress of the broker and then each trigger.

A

But the interesting thing is that, because we are writing a controller for the auto trigger the consuming service, that at is asking for a filter for triggers. We we know who it is so we can set its own irreverence, and so we can get automatic garbage collection all right. So this this is, it's actually hosted Mackay native service running in a cluster, it's dynamic. So every time you load, the page, it'll read query.

A

So if we reload real quick this week, so graph actually uses three duck types that we have in K native and one of them is it understands the general shape of it we use for status, so we can see arbitrary, ready conditions. I can look at that and understand how to color the graph to give you hints about.

A

What's going on so they've been ready, so we added a new function that does prime classification and if you would guess it, it then filters events that come through it, labels things that are prime and it shows it upon the saw the Sakai service, that's, prime, so yeah.

B

What you do as well questions and we're gonna put up a wall of links.

D

Thank you so much. It's an open mic ends up over there and yes, I'm a runner. The.

A

First question is: we have produce.

A

We've produced these stickers so yeah, so the duck hasn't been there.

E

This looks really cool. How I'm actually curious how you're determining shapes? Are you actually looking at like the CDE specification and like the air pas API spec? Are you doing something at the go layer like where is that happening a good.

B

Question so right now we're not doing anything based on the open, API schema. Although that's an interesting thing we thought about. We actually don't publish the open, API spec in the key native resources yet, but.

A

We yeah, we only do partial schema from the parent down so yeah, so it needs that one leaf and all the way down. So you can do merging so.

B

For the the second half, we basically leverage labeling right. So, if c or DS that are labeled, saying I am addressable or I am a source or any of a variety of different duck types. You can basically reason reason about those resources with the expectation they implement that shape. They have that common partial schema for the Sikh bindings. Basically, it's somewhat on the user. There isn't really a great way to validate this right now, or at least we haven't come up with one. If you have ideas, let us know the schema.

B

One is definitely an interesting one that we haven't fully explored, but yeah. So if you reference a pod spectable thing in the target of the sync binding I mean ultimately, if we try and start patching that or if it goes through the mutating web hook, and we try and apply a patch that we can't apply, it'll actually get rejected, assuming there's good validation after the mutating weapon runs. So hopefully that answers the question in both contexts, if not happy to keep discussing.

F

So when you did the cron job, the cron job wasn't what was actually injected. That was the job that it created through the mid head in my book. What, if the thing that you wanted to actually work inject was the cron job which doesn't have a templates back at the top, but does two levels down? Have you thought about where things they had the type, but it's just not at the same path, yeah. So.

B

You know yeah cron job actually is not pods peccable. It has some extra layers in there but yeah. So if you wanted to do it with a cron job, it would either have to be well.

B

You would either have to start teaching it about multiple shapes and a way to disambiguate when you use one shape or one to use another shape or.

B

Yeah I mean that's mainly it right, so one of the one of them.

B

One of the things that we were sort of fiddling around with and kind of wanted when we sort of figured out that we could do this. The way we're doing it with duck typing was the idea of custom sub resources, so kubernetes has, for instance, the status of resource the scale sub resource, but for c r-- DS, there's really only those two I think there may be a third that I'm blanking on, but you certainly can't create your own.

B

So if you could effectively say so when I'm defining my scale sub resource, for instance, I tell kubernetes where to extract the various properties of the scale sub resource of the selector. The replicas count the desired an actual replica count, but if we could do that effectively for where in my object is the pods back, that would be pretty cool and we could potentially do something more powerful. But then again, that's sort of like having some higher-level information could use to disambiguate and guide how we're doing it for various types.

B

A

Like all things, we are starting small, we're building on condoms. We we know- and so now we have this if the shape is exactly that, what we expect, but there's other noise around it. We can deal with that. We could probably work now that we have this. We can also add by the way you can look in a different half. It seems reasonable.

A

All the way in the back, you.

D

B

Purpose after this, it's it's your turn, no more than here.

C

Thanks for so for the run, so any reason why you chose shapes for matching and not some kind of expression, language such as XPath.

A

We're trying to solve the problem is for venting, specifically in the the reason. The first duck typing talk happened. We have all these consumers those consuming functions, but that consuming function could be any other thing, so custom resources as long as they expose how to talk to them. We can wire things together without actually understanding, so we've we've kind of started and focused on.

A

It must be this path and also there's a lot of there's a nuri inside of kubernetes to give us the unstructured object and then give it a JSON and it marshals it in, and we can deal with that. So that's that's. Basically why yeah.

B

I'll give you a shorter answer at doing duck. Typing like this is I think slightly harder than just having fixed types, every we're taking it the next level and having it be a bit more dynamic. That's really really hard. You know it may be something interesting to explore, but I mean where we are for some of the shapes we've had to reason about same shapes been pretty effective, partly because of API consistent, seeing the API principles and kubernetes right. So one of the duck types is status. Conditions have a pretty regular shape across kubernetes resources.

B

All of the apps resources did a really good job of consistency across that class and there's.

A

Some other, like more practical examples where everyone tends to type the kind, but kubernetes doesn't operate on the kind it it operates on the resource which is pluralized. The pluralization operates on english and that's broken right so to convert from kind to resource. You can actually go and query the keys or have a have a list or whatever. So that's kind of the first iteration of what started some of the polymorphic parts.

D

Okay, we have time for one short question and short answer. Anyone back.

G

When you're building controllers that are like doing this type of duck typing, do you have any thought towards actually being able to wire in, like you know, informers and things like that for the subtypes that they actually generate as well? So.

A

We we actually have typed informers, okay,.

F

A

Can you set it up slightly different? You have to give it a gbk so that you can actually make that correct, right, yeah, I call, but then the controller gets back the duct type thing, so it reduces a step for us. Okay,.

G

What I was thinking is like from that top level type there gonna be other objects that are other types that are generated by the controller, that's managing that other type, and you might also want to watch those like sub objects, a generator I.

A

G

A

Think that's the next step, yeah doing this, the the preparation for this demo. We kind of we ran into a problem of great. We understand some of the the dag in the data plane, but we we start to need to understand the ownerships. For that exact reason. We don't know what resource just wants to be deleted because a child was deleted or if the deploy made a replica set right.

B

So I think I think one of the interesting things in this in this example, so I think correct me if I'm wrong, but we've got the CID controller and that's your D controller is going to stamp out the duct piped controllers and so with what you're talking about is effectively if those duct these these instances down here, blue bar Baz, those are creating resources, have it basically having informers for those resources? Yes, so so yeah, so I would expect that, generally, whatever reconciliation process. You're writing.

B

You are maybe taking this sort of abstraction of the concrete types that the booth of are the bars and your reasoning about it. In terms of say, what images are there and the types are actually instantiating are likely concrete types. So before you even stamp out blue bar and Baz, you can actually start up Informers for those and then, as you stamp out the duct typed controllers, they can basically leverage the the existing shared and former that was started up and then just sort of pick up from that. Does that make sense?

B

D

They said yes,.

B

It made sense: okay,.

D

Thank you so much for joining in Scott and Matt run on a post.