Cloud Native Computing Foundation ServiceMeshCon NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Writing Service Mesh Control Planes in Rust - Eliza Weisman, Buoyant, Inc

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Writing Service Mesh Control Planes in Rust - Eliza Weisman, Buoyant, Inc

In this session, Linkerd maintainer Eliza Weisman will discuss the Linkerd team's experience using Rust, why they chose it for their data plane, and, most recently, how Linkerd has extended the use of Rust into the control plane as well. The Rust programming language has rapidly grown in popularity. It offers several features that help write reliable, fault-tolerant, and efficient software — all desirable properties for a Kubernetes controller. Linkerd, the graduated CNCF service mesh, has been using Rust for its data plane proxies since the release of Linkerd 2 in 2018. The data plane has to be as fast and secure as possible, so Rust was a natural choice. However, like much of the Kubernetes ecosystem, the Linkerd control plane — which manages the behavior of the data plane — has generally been implemented in Go. Linkerd 2.11 introduced the new policy controller, Linkerd's first control plane component implemented in Rust. Join this session as Eliza shares the team's challenges, benefits, and lessons learned using the Rust.

A

Okay, so uh my name is Eliza Weissman I'm, a Linker D maintainer I work for buoyant the creators of linkerty if you're a fan of linkerty you've, probably seen me on the lincrity slacker on GitHub or on my Twitter account, where I post pictures of my cats and bad computer jokes.

A

So I'm going to be talking about um writing service mesh controllers in Rust. This is my rust. Evangelism slide, I'm not going to talk too much about it at this point, I think a lot of folks have probably at least heard of the language, but it's sort of core value prop is that rust is a systems programming language that ensures memory safety, so no dangling pointers or use after freeze, but it ensures this at compile time through compile time checking rather than at runtime with garbage collection.

A

um But this is service mesh con. So it's not roscom right. We're going to talk about a service mesh and the service mesh that we're talking about is Linker d.

A

uh It's a cncf graduated project, Ultra lightweight Ultra, fast Security, First, Service mesh for kubernetes uh and in particular the Linker D team's goals are that um the mesh should be secure, resource, efficient and low latency, and we really care about those things in that order right that security is the most important and then performance and efficiency and down here we have Linker linky the lobster the Lincoln mascot.

A

uh So, oh so linkerty loves rust uh the Linker D2 data plane. The proxies have been written in Rust since the release of linker D2 in 2018, uh but the control plane, on the other hand, uh is mostly written in go this year's Linker d211 release, though we introduced a new control plane component, the policy controller, which was written in Rust, which is what we're going to be taking a little look at today uh down here, Ferris the crab, the rust mascot linky, the lobster they're both Crustaceans, so you know match made in heaven.

A

uh So since the data plane handles all the network traffic in the mesh right, you really need those proxies to be secure. Efficient and low.

A

Latency low latency means that we really want to avoid garbage collected languages like go or Java, because a garbage collector pause impacts tail latency significantly, so you don't really get that predictable performance of a language that isn't garbage collected like roster C plus, but we also really don't want to deal with the kind of memory on safety related security issues that you get from C or C, plus plus we we don't really want um Heap corruption and use after freeze, so rust was really the only language we felt was an acceptable choice for the data plane in Linker D.

A

um This isn't really true in the control point. We need it to be secure and reliable, but the performance requirements. Although we care about performance, the performance requirements aren't really as strict as the data plane. The occasional GC pause isn't that bad, so languages with managed runtimes like go uh totally suitable and most of the Linker d control plane is written in go. So why did we write the new policy controller in Rust? Well, we liked it.

A

The linkage team has invested pretty heavily in Rust we're big fans of the language we wanted to use the same language or the same libraries we use in proxy and rust has some benefits in the control plane. We like we like a lot of its language, features that make it an expressive, productive language that also help us to write a correct code and help us to make invalid States unrepresentable.

A

uh So what's the link to d control plane actually doing well, to put it broadly, it's a set of microservices the Run kubernetes cluster and they watch resources from the cube API, and then they serve eight like services and pods, and then they serve a grpc API, uh that's consumed by the data plane proxies and the proxies use those apis to discover where to Route traffic and what policies to apply when routing that traffic, in particular.

A

This policy controller that we're talking about today is responsible for telling proxies how to enforce server-side policy, and it's watching a set of crds that describe policies like what clients are authorized and how to authenticate those clients, and then it tells the proxies that make streaming. Grpc requests that controller, uh how to apply those proxies.

A

So if we want to write controllers in Rust, we need to talk to the Kube API, so we need a set of bindings. Fortunately that exists now this is creates dot IO. This is the rust package registry, and this is a package called kube.rs and it's kind of like a rest version of client go.

A

uh It also has some other things that we might get to see some of shortly and then on top of coop.rs, the Linker D team wrote a little Library called kubert, uh so it's like Qbert, but with a K, it stands for Kube, runtime, Cube RT, and it's an opinionated runtime for writing. Kubernetes controllers in Rust. Here's some of the apis it provides. It has utilities for things like running an admin, server, doing graceful, shutdown and logging, and so on.

A

We're not going to look at that in too much detail, but what we are interested in is that we have this module called index which provides utilities for indexing kubernetes resources. So this is um the trade index namespaced resource from kubert's index module. A trade is like a rust interface that we can Implement on our own types.

A

This is generic over some T, which is a type that represents the resources that we want to watch, and then we have three methods on this trait apply is is called when the kubernetes API creates or updates a resource.

A

Deplete is called when a resources is deleted and then reset is called when we have a reset event and a large number of resources change at the same time, so we can have some Rust type that we defined and we implement this trait, and when we implement this trade for our type, we can give it to the kubert. Runtime and kubert will just call these methods, as various resources are created and deleted, so those resources could be core resources or crds. In this case, we care about both crds and core resources.

A

So how do we actually create crd bindings? Well, here's an example of one of the policy crds the server resource. This is different by linkerty and it defines a server that receives traffic in the mesh and can be assigned server-side policies. So here we have a server that lives in the Linker dvis namespace. It's named admin. This actually came from a real link to the install and it has a spec with a pod selector.

A

It matches a set of labels on pods and to say these positive part of that server, and then it defines a name Target Port on those pods, and then it can include a protocol hint. So here's how we actually Define that resource in our rust code. So we have a bunch of these attributes with the hash on this res on this rust, struct type and those are actually in those are derived attributes right.

A

So they're saying we want to implement these traits for this type and there's some code generation that we're calling into essentially that automatically provides those trade implementations. So if we want this to be a binding for a crd, we say well, we need to be able to deserialize and serialize it and we need to be able to generate a Json schema and they're a library called survey.

A

Just lets us put those attributes on this type and all of the serialization deserialization code is automatically generated, and then we have an attribute saying well, we want to rename the field so that they're camel cased instead of written in the the rust case with an underscore, and then we have these extra attributes um like that's rename all, and then we have here a derived, Kube custom resource.

A

That's saying this is a crd binding and then this attribute here is saying well, this is the group and version and then the name of the resource, and it is namespaced and that generates the entire crd binding for us. We don't actually have to write any of the code for parsing, the Json and, and you know turning it into this in-memory representation.

A

So, finally, here's an example of how we sort of would wire some of this up. This is a main function. You know in a sort of example, rest project. This isn't really from our policy controller, I've, stripped it down a lot to sort of not put so much code on the slide that we can't read it. But this is an async main function: it's using Tokyo rest sort of leading asynchronous runtime. That's not super important, so we we parse command line.

A

Arguments and kubert gives us some utilities for taking CLI args like configuring, the Kube client, here's the Kube context you want to use and so on, and so we have this Arch type. We parse that we get the client configuration, we configure the runtime with this Builder, okay and and then here we start, we actually say: okay, we have this index type that we've defined. So we want to make an instance of that and then we want to have the runtime start, a watch on This Server resource.

A

That's the type here we could pass in list parameters like we want to only list these namespaces or filter in this label selector. But we're not doing that because we want everything and then we say: okay kuber.

A

We want you to actually index um in the namespace resource using that index namespace resource trait that we just looked at, which we're assuming is implemented for our index type, and we want to use this watch on servers and we could do this for the same index type with all the different resources that we care about, and we can establish like relationships between different resources in that Index this way, and that index is just whatever data structure we want it to be, it might be like a nested set of hash, Maps or something it might have different relationships between the different resources that we've indexed and then you know we can do other work here.

A

Kubert also has a utility for running like a validating admission web hook, which we might start here. We might start the grpc servers that we use to serve the proxy API and so on, and then, in the end we just tell the runtime okay, we actually want to run this and it will run all of these watches that we've started and we'll call into the different methods on the index and that's really the whole thing.

A

So this code is actually live right now in linkerty, if you're running, Linker, D you're running code like this, you can check out the policy controller on GitHub and we, you know, are running a service mesh Academy. You can sign up for if you're interested in linkready and learning more, we have Hands-On workshops for production users. That's free! We also have a managed Linker D, offering that you can check out at the buoyant booth and the of the vendor.

A

Hall and uh I'll be hanging out at the Linker de booth in the product Pavilion for most of the rest of kubecon. If you want to chat about this stuff or ask me questions about rust or Linker d, that's about it! Thanks for coming.

B

Eliza did a pretty good job on timing, so we have some time for one question: if you have yeah yeah any questions. Yes,.

B

How did you said most of the control players written in go? Currently? How did you handle the interoperation interoperability between the go and rust code, or are they running as separate binaries?

B

There is another microphone you can use. Oh okay yeah. They should be. You know, yeah yeah,.

A

Okay, so that yeah that's a great question: um they are running in separate binaries uh they're, just they're. Essentially the control plane and linkage is essentially a set of microservices.

A

We have a deployment that deploys pods that have all these different control, Point Services uh and in general, the way that the Linker d control plane, Services, communicate with each other right is just through kubernetes, so they create and update and delete resources, add labels to resources so on, and that allows the different controllers that do different things to share State uh they're, just kubernetes controllers in particular. Some of them also serve apis. The data plane consumes, in addition to just interacting through Cube resources,.

B

Thank you so much I think the apis basically solve the problems in kubernetes. Api solve the problem of inter communication. So uh let's do one more time for Eliza.