Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Life of a Kubernetes API Request by Daniel Smith, Google

Description

Life of a Kubernetes API Request - Daniel Smith, Google

We'll follow an API request through the Kubernetes API stack, starting from kubectl and ending with the storage layer (etcd). This is targeted at people who want to develop Kubernetes itself. We will look at code snippets! (This is a talk I have given internally at Google.)

About Daniel Smith (lavalamp)
One of the first Kubernetes team members, I currently lead the Kubernetes API Machinery team within Google.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello: everyone, I'm daniel smith, I'm, a software engineer at Google. I was one of the first individuals on the kubernetes team back before we open sourced it before we released it.

A

So this talk is something that I did at Google. Actually, it was two parts: I did it for new team members.

A

If, if you've ever been through the Google new employee orientation, they've got a series of classes, they a lifetime of a server lifetime of a query. All these things so I thought it would be good for new team members just to see a lifetime of a kubernetes api request, and apparently all of you agree. I thought there'd be like five people interested in this.

A

So the the question that we're going to try and answer here is what happens under the hood when you start up a queue patrol command specifically I picked replica sets because they're, you know relatively simple. So what happens when you type cube, which will create ah and some replica set IMO and it turns out, there's actually quite a lot that happens so I had to cut out some stuff to fit this into the time, so I'm sure I cut out your favorite part.

A

Sorry, so these are the these are the things that we're going to go through is what happens inside the cube control binary.

A

Then we're going to have a short interlude on what happens on the wire conversion, serialization defaulting, then we're going to go through what happens once API server receives a query and then last we're going to go through what does API server do after it's received a query, so those are those are the four things that hopefully we'll get through.

A

So this is this. This talk of sort of aimed at people who are interested in the nice codebase, so I'm going to actually have code on my slides I. Don't expect you to read the slides as they're up there, I'll try and point out relevant things, the the slides and the recordings the feedback I got from the talks I did inside Google is that people like to watch watch this and sort of a go. Look at the code.

A

You know pause it occasionally and go go browse of the code, so I have included on these straight on these slides. All the all the function names are links, you can click there. It'll.

A

Take you to the one point, four point: five version of kubernetes- and you can you can just sort of use this as your guide to the stack so treat this as reference material, the code to fit to fit anything meaningful on the slides, I had to basically delete mostly everything so take everything you see in the slides, with a grain of salt click on the link before you trust anything.

A

So with all that said, let's, let's dive in so the first thing that happens when you run Q control is that it builds all of its available commands.

A

You know this happens every time. You start it up. We use the the app we use the Cobra library. The output of this process is documentation and, and also commands that can be executed. So I have here on the screen the setup code that that creates the the create command, not too special you'll, see down at the bottom. The run create function, call I, don't know if you can read the slide or not, but the run create function. Call is what does all the? What does all the work which I have on the next slide?

A

So if you could read the slide, I think it's a little fuzzy, but if you could read it, you would see that it invokes something called a factory. It makes a builder and it makes a helper. What are these things? These are? These are the three like components of cube control. If you do anything involving a cube, control command, you're, probably going to interact with these, so I'm going to run through what each of them what each of them is. First up is the factory, and this is like a replaceable logic for cube control.

A

I know, openshift overrides it in open shifts command line client, so it's basically just drop in things like one of the function gives you a client that talks to kubernetes server. The the one that we'll look into further is the unstructured object, which gives you a just mapping between an object, type and a rest path on the server.

A

This is used to do generic generic rest operations, which is what we're going to show in a minute here. So this is this is the factory, and there is a there is a function linked here which builds the factory is like 600 lines of code I counted. It is just full of like function, definitions and again, openshift overrides this. This is this is drop in logic or for Q control yeah.

A

So next up on our list is the Builder and I. You know I didn't name these things in queue control and by the way, this talk is more of a history lesson than it is like. This is the way I think things should should be. We started off with a lot of little bits of code and and over time they grew and they grew layers and the layers grew layers queue. Patrol, isn't that bad? This builder thing is actually necessary.

A

The the task that it performs is is all your when you invoke queue control you have a like a f command or you can pipe in standard in or you can give it a directory or you can give it a file with a bunch of resources in the builder is responsible for unpacking that and turning it into a list of resources to iterate over yeah. So you can, you can go browse the coda fed.

A

If that interests you and finally, the last part of queue patrol that we're going to look at is the the helper, and this is responsible for for like generically taking some resource and doing some rest thing to it. So, in our case, it's taking a replica set and it's calling the the create method you can see at the bottom here it I, don't know.

A

If you can't see if you could see at the bottom there, it has a chain of commands like post, name space resource body, do get that's interacting with the the next layer down which I think is on the next slide. Yes, the next layer down so now. At this point, we've exited queue control we're in the the kubernetes client library, which is which is now published separately, and you can use this code in your own projects and in fact the links on this slide.

A

Go to the client go library and not not the main kubernetes repo. You can see. The rest client here is basically just a helper tool that helps you build a resource path, a URL that refers to a kubernetes resource. It's got a series of helper functions, so you give it a verb. You can give it a prefix or suffix. A resource like like replica sets, in our case a sub resource, the name of your object, if you're doing like a gat, a namespace yeah. So this is how you build up a request.

A

One thing I wanted to hit on in particular is the body method. I think it's kind of nice lets you pass in the body and multiple formats, so you can pass a string directly. You can pass an array of bytes.

A

You can pass a the IO dot reader, which is probably what you want to do if you're following normal ordinary go conventions or you can pass in a runtime object and and runtime object is really the the type that we've made in the kubernetes code base to represent kubernetes api objects yeah, so in it and it does the sort of does the right thing. One thing one thing to look: is this this runtime in code?

A

There I don't know if you can see that is calling down into the serialization machinery so that that is the place where the the magic happens of taking some object, that you have in memory and turning it into bytes that you send over the wire okay we'll get to that in a second. Finally, once you once you've constructed a request, you call this du method. It actually goes out and executes an HTTP request.

A

I didn't put this function on a slide, but if you're interested, you can go see the the code where we actually make the the go HTTP client and execute a request, and then on the on the read once it gets a response back. It transforms that which, which might involve turning a blob of JSON into a go error, type sending that up up to the user.

A

Okay. So so that was the cue control side of things and we've gotten bytes out onto the wire at this point, and so I thought it might be good to talk about what is we've got a if you've looked at our code base. You know you know that there's a bunch of automatically generated code that does a bunch of strange things and I, don't think we have time to go through all all the things that this code does, but I wanted to sort of.

A

Tell you what the what the problems are, that this code is solving, and why do we have this complicated machinery, so the basic problem for serialization is we have we have some memory structure? We have to have some canonical way to put it on the wire. Otherwise we can't share with anybody for conversion. Why do we have conversion?

A

We have old clients and kubernetes is infrastructure people right, important clients against it. People right important clients that work with. You know to two or three or four older versions of kubernetes, and we want those clients to continue working. Otherwise people are not going to be able to upgrade so so that's that's why we have some conversion machinery I'm going to talk more about these in the next slides.

A

And lastly, why defaulting the reason why it's here is because we implemented it it in the same way that we implemented the other two, but the basic problem is there are API. There are fields in their API objects which have default values and, and sometimes these fields have you know some small number of values that are actually acceptable and old clients.

A

Don't know what those fields are so so somebody has to default them yeah, so so, serialization in Cooper Nettie's, we serialize to JSON primarily and Clayton, and some other of our friends at Red Hat have written a proto serializer. This is not the standard protocell réaliser.

A

We needed to maintain compatibility with our code base, so so, as reacts actually really impressive. He's got a thing that generates a proto message from the ghost trucks. It's pretty cool. It's more generated code, though, for JSON we use a tool called Google G, which generates decoders for us.

A

Although Tim was just telling me earlier today that he's working on writing a replacement, gorgey runs really slowly, I, don't know. If you ever rerun our generated code, it takes a while.

A

So I thought I would show you not not the machinery itself, but just some of the interfaces that we require objects to conform to this is the runtime object interface. So all of our API objects implement this interface a little bit of trivia for the longest time this this this interface had one function which was called like. He is a runtime object. It didn't do anything. It was just there to mark to let you mark certain structures, runtime objects and let the machinery operate on them.

A

Eventually, we change this to get object kind which performs a more useful function and replaces a bunch of machinery that use the reflect package goes reflect package, so this is much faster.

A

We have because we support both Jason and proto. We have to have an interface around our encoding and decoding machinery, because you could have a blob that is proto or or JSON you need it. You need your code to work with both, so this here I've shown the the encoder interface, the decoder interface and the serializer interface, which is just the two of them together.

A

Let's see I figure out what it's going to say, yeah, so you have a blob, you don't know if it is proto or json. We have an interface to manufacture serializer for you, oh I, remember what I was going to say: I'll state on the next slide, the so the negotiated, serializer interface will give you back a serializer for if it's particular media type. So this goes a little bit beyond just proto versus json. It's also.

A

It also deals with encoding. So there's a content, type header that sort of thing we're going to add more in the future, where clients, because we'll be able to set the accept header and get back results in a format, that's more to their liking. So then this is the mechanism that we'll be using.

A

So at the bottom of this you see encoder 2 version and decoder 2 version. That is where the that is where the conversion machinery comes in so like I, said, the goal of the conversion. Machinery is clients, speak old versions. We want to keep those clients working.

A

This is this is really kind of why we ended up with a weird conversion machinery, a weird API machinery- I love this like. If we, if we did this again, what we do, the same thing, I, don't know- maybe GRP see- did not exist when we started this, so that wasn't an option we haven't switched to G RPC for a number of reasons. One is we have existing clients that expect this now and it's hard. We can't break them yeah it's complicated.

A

So what I was going to say is that proto itself does not have a great system for keeping old clients working. There are some rules. You can follow to keep your proto messages compatible backwards compatible, but it works best in a situation where you're, like rolling continuously rolling out of service doesn't work great. When you have clients that can be an arbitrary arbitrarily, far number of versions behind you yeah, especially if you want those clients to continue to work, as opposed to like just not crash. So so we came up with this machinery.

A

It works, it's not pretty yeah. So our solution, basically is we'll just keep the older versions in entry and and we write, we actually automatically generate some functions that convert between the old and the new, the new format.

A

So in terms of what that looks like in code in terms of what that looks like in code, it's actually exactly the same interface as the serializer.

A

It just has different semantics, so the serializer is not going to manipulate the data anyway. In any way, it's just going to take it from wire format to memory format. The the codec may additionally transform the data in some way if I turn turning it from like. If it's view, one on the wire, maybe it'll be V two in memory, or we have this concept of an internal version which is like the the hub of the conversion diagram.

A

So all versions can be turned into the internal version and the logic and the server can be written understanding. Only the internal version.

A

Finally, defaulting like I said: clients may not understand new API fields. Solution is the server ads default values for fields until recently. Actually Cube control was unfortunately adding default values using the same machinery because it's built into both the server and queue control, but we've taken it out of I, think create and update at least and we'd like to take it out of more places and keep control. Because really you want the server to be applying the defaults, because the server has the best idea of what the defaults should be.

A

Little trivia about the defaulting is that it is applied, recursively and so currently fields are defaulted multiple times. This is a bug unless somebody's fixed it I've been out for a few weeks on paternity leave. Hopefully somebody's fixed it, okay, so enough about conversion, let's run through API servers, processing of requests, so I've started here with the the run function in our generic API server library, we're making efforts to separate out non kubernetes specific things into this generic API server library and keep the kubernetes specific things in just the plain old API server package.

A

There's a number of reasons for this which I won't go into, but basically we want to make it easy for people to build kubernetes like a pis, and we don't expect you to write this from scratch. We'd like to you know, there's a bunch of stuff that can be reused. So this is the run method, and this slide is indistinct, but at this at the top it adds some.

A

What is it add? It adds some rate-limiting there's a timeout handler. It adds a rate limiter, timeout enforcer to the request, and then here at the bottom it actually calls listen and serve TLS. So this is sort of the entry point for the for the for the socket where it starts listening and serving let's see so so then this this code is all happening at setup. Time like when you start the API server binary.

A

This function is initializing the handler stack so.

A

When requests come in to API server, something accepts the request. It goes through. This stack of the go Hana HTTP handler interfaces that goes through the stack of them and each one like check something and then passes it along to the next to the next layer. So a as you can see like the the list, the the bulleted list is actually the opposite order of the the order they appear in code. This is because each this is because each each handler wraps the previous handler, so they actually get invoked in the opposite.

A

Adder order that they're added I think there's actually a bug on this slide. I think we're calling authentication and cores in the in the wrong order and.

A

Yeah so after it gets through so so the so the handlers are timeout. It adds a context to go through a content occation, it checks cores, it adds the logging, the logging handler and then finally, it goes through a series of dispatch operations. First, it goes through the muck, sir, that the go HTTP library gives us and then, unless that's one of the there's, only a few URLs that that handles and then it is otherwise passed through the the markzware that we set up with go restful.

A

So how do we? How do we set up the Luxor that go restful, uses I'm glad you asked this? Is the API group version instruct our API is divided up by group? It's divided up in other a ways, but the important thing for setup is that it's divided by group. So each group has one of these struck spelled out for it and then we pass it along to the API installer, and so you can see that we put in a bunch of helper things. So there's a negotiated serializer, that's passed in there.

A

So before creating this, you have to set up a serializer that can understand all the various formats that you want. You want this to use and there's also a storage. That's the it's yeah way back in the day, it was called storage because it was actually storage. There's been it's grown, a few layers since then, and now this is actually the the thing that handles the the rest requests, the business logic, which you know it wraps an actual thing. That is really the represents the storage which we'll get to in a few slides but yeah.

A

So it has so has all this. It has an object, I per creator, converter, linker and a bunch of other things, so that all gets tossed into the API installer, which starts off innocently.

A

Looking, it's got this install function and it just runs through all the groups and it calls call some swagger an OPA open API set up code so that we get nice API documentation as its building the, and then it calls this register resource handlers function and, unfortunately, if you drill into that function, you'll find that it is very, very, very long and not easy to follow.

A

But basically it's set Sappho a la a long list of handler paths that it then programs in to go restful and I've excerpted on to this slide, only the only the create one, but just just imagine this code copied and pasted like eighteen times for all the different handlers. There's a big switch inside this four loop yeah so, but that that I don't know if you've ever used go restful before. But that is programming go restful. It adds it adds, tells it what what verb it is.

A

It tells it where to route it to this handler, which it gets from this create resource function, I, guess to the documentation it with some parameters, you know and then then it finally installs it yeah. So that's that's. How we program go restful, okay, so so, finally, we're getting to something that is happening not at run not at initialization time, but actually at run time when a request is received. If you look at the this create handler this internal function, you can see it's. It's actually doing a bunch of interesting things.

A

It does version of format negotiation. It calls that it calls that serializer. It reads the body it does the decoding and the conversion to a specific version. Then. Finally, it calls into the I call data registry. It's that's the storage thing, the business logic after that is done. It sets a self link on the result and it sends it back to the user.

A

A

Let's go down a layer into the next set of the next set of layers, which is the storage interface, and originally it was one interface, but it's grown into a bunch, an interface per operation, basically so I'm only showing you the Creator interface. So, basically, when you define your object, you can define like just a create method of like this, this signature and if you send it through the API installer machinery, you will support the post verb and we have a bunch of different interfaces, one for each verb and some for various sub resources.

A

Yeah. So, since that's a lot of work to implement, we have a implementation ready-made for you, so all you have to do is is supply, something that implements the this strategy and most of the work is done for you and just to show you how well okay. So this is the work that is done for you.

A

This is the create function. So when you create anything, it runs through this code at the top it calls before create it does some stuff for you, that's the same for all objects that calls create. It finally does after create make sense, so when you go to implement a strategy, this is the replica set strategy.

A

You actually don't have to write too much code, so here it is setting the generation to one. We could be here for another hour-long session to explain the generation, so we will gloss over that. I also showed the validate function. Okay,.

A

So what we just saw was sort of the user-facing half of the API server stack. Now we're going to show the storage facing half this API server. Those are the those are the two main things that API server talks to is users and storage. So we have to store. We have to store things, so this is the interface that all of our business logic expects from a store. So this is and I didn't show at all. Does this got create delete, watch, get list, I think there's a update or set yeah.

A

So this is the interface we have in the code base. Etsy d2 and Etsy D 3 implementations I'm, showing links here to the cd3 implementation, but the Etsy d21 is right next to it.

A

So it's easy to find this is kind of critical code like you, don't want to write something to kubernetes and have it vanish off into the ether and the other reason why it's critical is, if you're watching it's really kind of important that if you list- and you pass the resource version back and you initiate a watch that you don't miss any updates. So this really needs to be tested very well, which is why I say on this slide.

A

We are open to interfaces for implementations, for more storage backends, but there's there's a there's, a high bar in terms of testing continuous testing scale testing. That needs to happen here which, which makes it difficult so I just thought I would throw that out there just for fun. This is the atv-3 implementation of create I, actually don't understand the at CD 3 code. So the point, the part that I wanted to point out is just it's using an embedded codec. It's called run time encode.

A

This is calling into the conversion machinery that I was talking about earlier, and this is because, when you start up a API server, you get to tell the system what version right, its storage and- and this lets you control like when you do storage format upgrades so so yeah. So we actually understand multiple versions, but we write out only in the specified version. This lets you do roll forward or roll back between versions so good to know. Okay. So how do we?

A

How do we get that codec into the storage pointer of we have written a factory to help you do that this code is in the generic API server and it actually works off of command line options that you pass to API server. So if you were to make your own API server, you would probably not need to mess with this.

A

And the the thing to know about the storage options that we take is that you can actually specify different resources, go to different Etsy d clusters, so by default events are stored in a separate at CDR default setup script store events in a separate Etsy d like you could do you can customize it arbitrarily. If you really wanted to like you could I don't know, maybe make secrets go to a different Etsy or something I, don't know, but that's just something you need to be aware of if you're messing with this code.

A

Ok, finally, I wanted to show sort of the glue that that calls all of these with all of these constructors and factories and wires it all up. So this this is also called install api's, but it's on the the master object and not the api installer, and this is where the this is, where the master package collects all of the different resources and sets up that group group resource struct. That I showed earlier and.

A

Yes, this is where it assembles the storage and and the the codex and all that stuff, and finally, I wanted to show like the specific code that sets up the replica set. Object is here and continues here. So this is this is where the business logic is constructed. Basically,.

A

Yes, you can see you can see here the customization of the various different parts of the strategy and I think that is, that is it. This was somewhat abbreviated from from the talk I gave at Google, so hopefully this was interesting useful if you think that there's one or two too many layers in this stack I would love for you to show up in the API machinery sig and help me make it better.

A

A

B

A

The self linker yeah.

B

A

All of our API objects have a metadata section. One of the pieces of that metadata is the self link which instructs the user like. If you just get the blob of data, how do you go find where it came from?

A

The self link tells you that, and so the it's part of the machinery to fill in the self link, otherwise everybody who implements their own resource would have to do that and if you've been around programmers for more than five minutes, you know that if you make them do the same thing twice, there will be bugs in at least one of those things.

C

A

What are the changes in 1.5 funny? You should ask I've actually been on a paternity leave for the last three weeks or so so I'm not entirely sure, what's making it into 1.5? This is my first day back, I I know we were planning on various improvements. I I, just I, don't know off. The top of my head would actually made it.

A

Yes, it's definitely going to improve mark.

A

Yeah in particular, I know, Clayton had a bunch of changes planned to support more formats in the negotiation. So, like the the stuff that you do, when you do cube cuddle get the the formatting. We were planning on moving that server side, so we'd like the server to implement that for you, so that would be an additional like content type.

B

D

Application actually reads an object and my question is: is there any plan holders and holders like eccentric library that I call, in my opinion, yeah.

A

So the question is: is there any plan to move encoders and decoders into a separate library so that they're easier to reuse? So we have moved the client into a separate library. I think the plan is to move more things like we would like the API machinery to basically grow into a few libraries. So yes, I would say that that's definitely in the in the in the future. So.

D

The right roads to using clients.

D

A

Yeah, so yeah right now, I would definitely recommend that you use the client library that we've published and that actually includes a fair amount of the encoders and decoders yeah, but I I think we're going to want to move the encoders and decoders in to get another library that can be consumed by both the server and client side.

A

Actually, we would like we would, like the conversion, part of that to be only linked into the server component, just because we want the clients to be simpler and and and because the server is taking on this, this effort of converting between versions, the clients really don't need to with the single exception of kube control convert, and we want to move that functionality into the server anyway, because if you imagine a world where you have multiple API servers and not necessarily all kubernetes, you can't really reasonably expect every single API resource type to be compiled into queue control.

A

So we want cute control to be able to call up to the server to get that done.

C

A

Is configurable through the question? Was the the different having different resources go to different Etsy D clusters? How is that configured that is configured through run time, flags that you pass to API server, so I can't promise that they're super well documented. But if you, if you dig around in there, you should be able to find them.

A

Yes, you can actually pass lists into into it. I forget what the exact format is. The help on the kind of relevant flag should tell you, but it's something something like you know: resource type equals you know yeah, and this a comma-separated yeah yeah are you believe, that's more than you should be packing into flags, but that's what we do.

E

A

This API machinery handle registration of new resource types, yeah I want to say yes like it's our goal for that to be easy and we have a another goal, which is which you should be able to implement a resource type in a in your own API server, and we should be able to combine those at runtime because right now, basically, you have to get stuff compiled into core if you want to test it out, but we would like to get to a world like you have third party resource right now, which gets you some functionality, but we would like to get you a little bit more functionality and let you reuse our components.

A

We there's a design out there for a there's design and like draft code for a discovery. Summarizer that'll actually merge multiple API servers and present that as a single thing to a client. So.

E

E

Right all us ah there's my certification recall.

E

Yes, but I can get okay.

B

A

So the question is running: running cube control proxy in a container the beta API groups don't seem to make it through. I can think of several things to check. I know: Q control proxy has some there's like some filters on it that control what exactly it allows through. So you might look at those flags to see if those are blocking it. If that doesn't fix it, I'm not sure file, an issue yeah.

B

A

The the I did not cover the authentication stack and in these slides, I think that could be another talk like even longer than this. One yeah we'd have to get David, EADS or Jordan Leggett. To do that, one.

A

C

And be using them, our shoe should.

A

We use api's that are marked as beta, so beta means that we promise it won't blow up the rest of your cluster if you turn it on, and it means that if you start using it, we are going to take on the work of keeping you compatible. So so we'll make it easier for you to upgrade.

A

So it's sort of up to you. If that is acceptable or not I can say for for gke Google's hosted kubernetes. We turn beta things on by default, so.

E

C

It's not cumulative what you get a Google. Thank you wanted.

A

Yeah, so so the question is generic API server? It's not kubernetes specific. Do we want to use it somewhere else? The use? We have a few use cases in mind. We would like OpenShift to be able to import generic api server directly instead of forking heaps tur, which is a component of kubernetes which does monitoring stuff. We would like them to be able to import it directly. I think that actually is working, yeah, okay, yeah and and the once we get those to target use cases sorted out.

A

Basically, the idea is anyone who wants to prototype or or even launch a in-house kubernetes api could use that and use the Federation machinery that I mentioned to to like merge it with the the base cluster yeah. So we want to run away for people to launch their own custom api's on top of the kubernetes cluster. We don't want you to have to modify the the cube API server, binary that we ship.

E

Reviewing all this that.

B

Is obvious? Yes,.

A

I'm not sure the question is: is there any? Is there on any obvious thing that we would do differently, I'm I'm, guessing that if we've started the project now, given that G RPC is out there, even though it doesn't support sort of the conversion case that we really wanted to support I'm guessing, we might end up using it. But that's that's a total guess, total guess. Yeah yeah I mean a lot of a lot of what we do is like. We are at Point a. We need to get to point B.

A

The market needs this thing built yesterday. We are incurring technical debt as we get there.

C

A

So the question is at CD: 3 supports GRP, see how do we talk to a CD 3 over G, RPC, yeah and.

A

There's a bunch of benefits for this we're going to start actually storing I I'm, depending on how much progress was made while I was on paternity leave, the goal was actually to start storing objects in proto format and 41.5, and we need at c d3 to do that, because otherwise a CD 2. We have to talk JSON to it, which means encoding, proto basic storing it. You know, that's very wasteful, so that's why we want to get to at CD 3. One of the reasons.

A

This question is binary, configure a fitted data at least three weeks ago, I recall seeing a PR to actually add binary, like, like a config map, has a map of string to string like to add a map string to byte array, yeah.

B

Would you recommend it for like maybe bath it might get update from hometown? That reason we will not use Google.

A

Definitely uses proto everywhere, yeah I, don't know if I'd make a recommendation without knowing a lot more detail right right. Thanks thanks, everybody.