Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The CRDs that Broke the Camel's Back - Alper Rifat Ulucinar, Upbound

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The CRDs that Broke the Camel's Back - Alper Rifat Ulucinar, Upbound

Custom resources are the preferred way to extend the K8s API server with a declarative API. They enable us to implement our very own control planes on top of K8s. K8s has performance guidelines and thoroughly investigated scalability thresholds but no guidelines for CRDs are available yet. Our initial attempts to install 1000s of CRDs revealed severe performance issues related to the API server, such as service disruptions and client-side throttling. And this further led to investigations to reveal the root causes of those issues. This talk aims to discuss how one can troubleshoot API server performance issues using profiling tools and to present some real world data that allowed us to pinpoint the root causes of the scaling issues that we initially hit. As the troubleshooting process is explained, the talk will also deliver some insights into the mechanics of CRDs. We would also like to share some tips in successfully getting changes into upstream and moving the ecosystem forward.

A

Thank you for coming to listen to my presentation so.

A

A

So in this talk, I'd like to you know give a brief introduction how we encounter these issues. uh So upon is one of the main maintainers of the crossplane project and I will talk about a little bit about cross plane and you know uh how we hit these issues and then I'd like to briefly mention. You know the client side see the scaling issues that we have hit and also I'd like to talk about the server side, issues that we have identified and you know how we identified them.

A

Specifically, you know the high resource utilization we have observed with the api server and how we profile the api server. What windings we have and also I'd like to briefly mention. You know how you can profile the api server. So it's really straightforward. So I'd like to you know, talk a little bit about it. It's helped us a lot in this respect and also you know I'd like to talk about. You know the kubernetes scalability dimensions and you know about crd scaling in that respect, and you know this will conclude to talk.

A

So uh apan is one of the you know core maintainers of the cross plane project in the cross. Plane project we are working on, you know, generating sometimes you know, custom resource definitions which correspond to actual cloud provider resources and, as you would imagine why this custom resource says you know you can manage the infrastructure so.

A

And, as you know, there are you know, hundreds of uh different resource types in its cloud provider and uh crossplane is a multi-vendor multi-platform project so that you know either your resources are in aws, azure or gcp.

A

You can have a cross-plain provider and many stores providers. This also, you know, means that in a single kubernetes cluster, you may want to install multiple. You know cross plane providers. So recently we have introduced a terajet, a cogeneration framework where you can generate. You know: crossplane providers, on top of.

A

Terraform providers.

A

This also means that in a single provider like we have generated provider jet aws jet, ager and gcp pro, we have these providers, and uh on top of these you know you can have hundreds of custom resources, and this is where we first observed our issues, because you know in a single, for example, provider jet aws, cross, plane provider.

A

We have over 700 custom resource definitions and in the other, azure and gcp providers, we have again hundreds of custom resource definitions and if you would like to install one of them- or you know multiple of them in a single, you know kubernetes cluster.

A

This means that you will have thousands of crds installed in a single kubernetes cluster, and you know this is how we, you know, entered this domain. uh This is where we started experiencing issues.

A

So when you sum up, you know we have about 2, 000, custom resource definitions,.

A

So we can categorize the issues in two broad groups. First, we have the client side issues that we will talk about, and you know we also have the server side issues. The first issue that we had observed was, you know very high, cpu utilization uh that also led us to profiling. The api server trying to understand. What's going on on there.

A

The upstream cross-plane community was already aware of you, know: high memory utilization.

A

And I will talk about them later in in the slides, so I will also briefly mention you know our experience with the managed kubernetes services and, let's start with the client side, throttling issues that we have run. So I think most of you, if you try to install you, know hundreds of custom resource definitions might have seen these error messages.

A

So when you run a ctl command, you know on a cold cache, so I will explain what I mean with a cold cache later. So please bear with me.

A

Sometimes you can observe you know some warning messages which says basically, which says that you know the request was throttled right. So you will see that the throttling was caused by client side. uh You know it's not related to api servers prior to and fairness.

A

You know floor control mechanisms, but it's on the client side. So.

A

If you run the same command, you know in the same peri in a you know a small period of time, uh then you will not be observing this. Most probably, it depends on the cube ctrl command that you run, uh but, for example, the cube city I'll get notes. Comment uh run this shaft shortly after the initial comment. uh We'll just return the notes so.

A

To explain this we will need to you know, talk a little bit about the discovery client, so the discovery client is responsible for, as you would imagine, the available apis in the api server right. So it discovers the api group list first so which group versions are available at the api server.

A

This also includes the, of course, as you would imagine, the resources that are part of the api server itself that are shipped with the core cross, plane distribution, but also the custom resource definitions and for each discovered group version. uh You know the discovery client has to discover the resources, the actual kinds under that group version.

A

So we have this discovery, client running, for example, as part of cuba, ctl or any other api server client it could be and we have on the right the api server. So the first request is, you know, down to the slash api endpoint, and you know we get a. We won that api group list response from that. You know listing the group versions available at that point and also uh this was the legacy endpoint.

A

So we also need to hit the slash api's endpoint and similarly we will get an api group list from that point, giving us the available group versions there and then discovery client needs to discover uh the kinds available at these group versions right. So this is done in parallel.

A

And from the discovery client, you see, you know some parallel requests uh for each group version that needs to be discovered.

A

So here, as you see you know, these represent multiple parallel requests and the first one, for example, discovers the you know batch v1 beta one and the second one xb1, for example, the improved version and the api service is of course, expected to return the api's resources documents available at those end points.

A

So what has been implemented is um you know the clients, discovery. Client itself is throttled without any feedback from the api server itself. So this is done. uh Why are a discovery? Client? You know why I am talking about rate limiter implementation currently and I'd like to briefly mention. You know how this is implemented to give some technical details.

A

So let's assume that we have a number of requests to do to the api server, but we have the token bucket rate limiter. You know as a throttler, so you can imagine this. You know, as a kind of bucket uh initially filled with uh you know some capacity, and this unit of capacity is represented by the tokens here and initially our tiny bucket here has you know an initial capacity of three requests or tokens. Let's say, and you know, there's another parameter uh which we call as the field rate.

A

uh So as requests are admitted, each request will take a token from this bucket. As we will see, and this you know bucket- has to be filled, refueled right, so the r parameter, the rate parameter denotes. You know the rate at which this bucket is filled.

A

So, let's assume that we have these requests is represented with a cube, but the time dimension is the y dimension, unconventional meaning that you know these requests are in parallel and initially we have three tokens in the bucket. This means that we will. The three of these requests will be admitted, meaning that will be sent to the api server.

A

But the remaining one request, because there is no token left in the bucket- will not be admitted, and you know it will be exponentially, backed off to be retried again.

A

So, as time passes, uh you know with the field rate the bucket is filled. We now we now have a token available in the bucket and when the timer for the you know, blocked request arrives.

A

It will be admitted this time and you know the error message that you see or the warning message. I should say that you see, you know, denotes how long the request was delayed because of this rate, limiting.

A

So the client core discovery, client uses a token bucket rate limiter uh with the initial burst, parameter of 300 queries or 300 tokens and the rate limit is, you know, 50.0 queries per second.

A

So these are the current parameters in kubernetes 1.24.

A

When we were tackling with these issues, we realized that you know in kubernetes 1.23 the parameters, the burst rate was 100.

A

Excuse me, the burst parameter was 100 queries and the rate was just uh you know, 5.0 uh queries per second, so this was basically due to a bug. You know these were not the intended parameters. The internet parameters were actually 4b 300 and for our 50.0 qps.

A

However, you know due to a configuration back, the actual parameters were hundreds and you know 5.0, which were uh you know, quite low when compared to the you know, number of crds that we were trying to install, and uh you know we saw lots of throttling because of this.

A

A

I'd like to also briefly mention about you know the cash discovery. Client, uh if you remember when I was showing a you know an example of the client-side throttling warning messages.

A

I mentioned about you know if you run this on a cold cache and we will see what that cache means here. So cubicity uses a disk.cache discovery, client which actually, as you would imagine you know, maintains a cache of the api resource list and you know the responses from the api server.

A

The discovery client gets from the api server, so this cache is maintained under user's home directory by default and it's specific to the api service hosting port, it's specific to the api server and, as you can see, if you are running an api server on the local host, you can see the ip address as the host name or you know. If the. If your config declares an api server with a hostname, then it will be under that hostname underscore the port number and you can see a directory structure of this cache.

A

So at the root level, under this cache directory, there will be the server groups that json file, which basically you know, holds the group versions available at the api server and you know for each api group. You will see specific versions available under that api group and under each group version folder, you will see the server resources.json file which actually contain the available resources under that group version so running the ctl examples that I that I've shown on a cold cache means you know running those commands on an empty cache like in that example.

A

I had just you know, run an rm command to you know clean the cache uh so that the discovery, client, the cache discovery. Client excuse me- would not return. You know responses from its cache, but this cache also, as you'll imagine, has a time to leave parameter uh up to kubernetes 1.23.

A

It was uh just 10 minutes, uh meaning that you know every 10 minutes. uh Even if you know the cash has already been populated, the discovery client would need to.

A

You know talk with the api server to fetch the latest versions of these documents. However, with kubernetes 1.24, it has been increased to six hours so the time to leave parameter for the you know, the default time to leave parameter for cash discovery. Client has been increased to six hours.

A

um So if you summarize, you know the throttling issues the discovery client needs to make. At least you know, two plus the group version count number of requests to discover the available apis right.

A

So the initial tool comes from the slash api and slash api's endpoints, and then for each group version. We will be requesting the available resources under that group version. So those sum up to two plus uh the group version count and also please remember that uh you will also, for example, if you are running a cube, ctrl get notes comment. It will also be a separate request that comes after the discovery phase.

A

So if we install, for example, all three cross-planed jet-based cross-plane providers, it means that we will have 300 370 group versions hitting the api server right. So what are the consequences? If we have? As you see here, the important parameter is not the you know, crd count, but rather the you know. Group version count right and um this table shows, for example, uh some dif. Some, you know, pyramid running. You know discovery client with some different parameters uh with cube ctl.

A

You cannot do this, but you know we have custom version of cube ctl through which you can explore these parameters and, as you would expect, since we are talking about 370 group versions and since the burst rate is by default. Excuse me, since the burst parameter is by default. You know 300.

A

Tokens just 300 queries, cube ctrl will be throttled if it's running on a cold cache right. But if you increase uh up to the initial burst, parameter up to 400.

A

Then it's not it's no longer throttled. So uh with this tool. If you pass the rate parameter as -1, the client-side rate through tooler is completely disabled, but we can see no improvement. You know after 400, because it's already larger than you know, 370.

A

So maybe we should also discuss you know the api, server cpu and memory high api server recipient memory, utilization issues that we have encountered.

A

So uh initially we were installing. You know 600 and you know about 700 custom resource definitions uh on a single cluster and I think it's not readable. But, as you see here.

A

The api server was immediately started. You know, spending two cores.

A

These are, you know, collected from prometheus metrics as you as you would imagine, and you know we investigated the also. Maybe I should talk about you know. uh We also saw high memory utilization, although we did not. We did not. You know profile. Do a memory heap profiling ourselves, we initially concentrated on the cpu utilization.

A

And you know I will first represent the results uh from our you know: cpu heap profiling. So the observation, the immediate observation uh from cpu profiling, was that you know uh over 40 percent of our cpu time was spent.

A

uh You know during open api, v2, spec, aggregated, open api view to spec serialization. So I will briefly mention what this means, and you know when you examine the available profiling data.

A

We spent api servers, spends lots of time in json on marshalling and marshalling operations uh and also you know, proto, uh binary civilization uh and also another interesting point was you know, because you know during this aggregation operation, uh lots of objects are being created and destroyed, so we had a very high hip churn, so the garbage collector also accounted for a large percentage of the sampled cpu time.

A

So let me talk about you know what this open api, v2 spec margin means uh with the v1 version of api extensions. Each crd must specify you know a validation schema.

A

You know expressed as an open api, v3 structural schema and using this schema, you know the a custom resource belonging to the custom resource definition is validated, as you would imagine during you know, creation and updates and unknown fields can be pruned by the api server that are not. You know available in this schema, and also this schema is used for client-side validation.

A

So when you run a cube, ctl command without specifying, for example, the validate equals false validates the ccr manifest, and you know if it doesn't conform to the manifest you know it will reject.

A

To work right so.

A

Api service api server serves an aggregated, open, api, v2 spec at the slash open api, slash, v2 endpoint, and this spec contains uh or documents. You know the complete api available at the api server. So uh by default, this is served as a you know, serialized as json.

A

So when you hit this end point without you know specifying uh with an accept header, a different serialization format like protobuf you will. The api server will return a json, uh but you know uh especially for intra cluster communication. You know a protobuf civilization has also been implemented in the api server.

A

So here you see an example.

A

Aggregated spec example, and you know under pets you will see, for example, the.

A

You know open api schema of a custom resource definition, so this also means that.

A

When you register a you know, custom resource definition, the api server has to recompute the aggregated spec right so that custom resource definition itself becomes part of what's available as an api in the api server and the you know, aggregated open api, v2 specs served at the slash open api, slash, v2 endpoint has to be updated with the new crd and, if you are registering you know a bunch of crds.

A

This also means that the api server will, in the background you know, update the aggregated, open api, v2 specs, so the profiling data had shown that you know this is where time was spent. So this was the root cause of the high cpu utilization that we had observed, and you know we got in touch with the upstream crossplay upstream kubernetes community and luckily learned that they were already available. Aware of you know, uh high memory utilization issues uh also related to open api.

A

V2 spec, uh you know, processing, and you know, a fix was on its way, so we have also evaluated uh so. I will also briefly mention what the fix was. So the fix was in fact you know.

A

Instead of calculating the aggregated, open api, v2 spec each time a crd was generated, the idea was, to you know, defer calculation of the aggregated spec till a request to the open api. Slash v2 endpoint hits right, so it basically, you know, defers the calculation of the aggregated spec and you know if you register a bunch of crds the open api, v2 spec is not immediately calculated and you know, whenever a request comes to the slash open api, slash, v2 endpoint, the api server will for once calculate the aggregated spec.

A

So this saves lots of you know cpu cycles, and you know we wanted to see how this affected our profiling data or whether we could quantify the improvements here, and you know. I think it's not that much readable. But sorry for that uh here you you'll, see you know the cpu utilization running with this fix.

A

So if you compare it the previous one, for you know about 30 minutes, it depends on the number of custom resource definitions.

A

You have installed with the provider, but the aps server was spending about two course for a prolonged period of time, and you know with the laser margining of the open api we do spec uh things are in a much better situation for a shorter period of time and with a you know, smaller peak the api server can handle uh the calculation of the aggregates to open api to spec.

A

So even with this fix, which is already available in 1.23, so we were not. We were hoping that these would resolve our issues, but we could not validate this in the manage kubernetes offerings.

A

This was because you know we could not run a modified version of cuba. Api server in these manage offerings, but after the fix was available we also revisited you know the situation in eks, aks and gk regional clusters- and you know this table shows our current situation so any eks by the way. This version does not contain this fix the laser marshaling fix, but uh you know we can also install all three providers successfully, uh but you will see the client-side throttling warning messages, issues and maybe even failures.

A

uh You know because requests may timeout due to client-side throttling, but whatever the reason from a cross-plane perspective, we could install all three providers and in aks clusters we were good with provider jet aws and provider jet asia, but you know it was not possible to install a church provider and in gk regional clusters we had some problems. You know in installing a single provider also.

A

So this also means that you know these issues are not resolved, and you know one of the purposes of this talk is as the cross playing community. We would like to share our findings and you know, drive the upstream community and other. uh You know pro providers uh solution providers who depend on large number of customer resource definitions.

A

uh You know to work with the upstream community to resolve these issues, so this is how we profile the api server. So we were using custom. Pro custom builds of cube api server, as you see, including the fix and not including to fix, and also to test some other ideas.

A

So uh you know this profiling data was collected on kind clusters. We first used kind, build node image.

A

You know running it on the specific kubernetes source repository that contains the cube api server to be built, and then using this node image you can create a kind cluster and then, after running cube, ctl proxy, you can hit the slash debug, slash, p, prof profile endpoint to collect profile data, for example. In this example, we are collecting profiling data cpu profiling data for uh 3 300 seconds and then using the you know.

A

Excellent p, prof tool- you can, you know, examine the profiling data, so it requires some experience to interpret the profiling data, but in our case uh we were able to accurately pinpoint the open api v2 spec marshalling uh you know as the root cause of icp utilization.

A

So um kubernetes is already you know a scalable threshold document.

A

It's maintained by, I believe this scalability community, but it does not consider the number of crds per cluster as a scalable to dimension, and you know the only thing that we could find about. You know the number of customer resource definitions, uh the limits for limiting. You know the num. uh The number of customer resource definitions per cluster was in the ga availability document.

A

uh I've put the references to those documents here and it's suggested 500 as a maximum limit for the scalable target. But, as I mentioned, this is not you know, part of the official scalable traditions document. So.

A

We also had some chats with the upstream maintainers and also they also believe that this has to be part of the scalable t documents, but you know it needs some time and you know they need to be aware of the use cases and you know use cases similar to ours.

A

So I'm concluding my talk. Thank you very much for listening uh here. You can find some further pointers, especially the top two ones from the cross playing project.

A

There are really detailed discussions and you know detailed, uh you know technical analysis etc that you might be interested in if you want to dive deeper into this topic and also the one pager has some.

A

You know good tools that you can use to generate custom, resource definitions etc for your tests, and you know for testing possible solutions.

A

Thank you very much for listening.

B

If anybody got a question any questions, let's take a look, any questions hands, not all at once. It's been a pretty long day, hasn't it lots and lots of action. uh I guess.

B

Is there anything else, you'd like to add regarding crds your experience with them, your experience before them, what we might expect from them in the future.

A

Yeah, so I mean is the cross playing community? We really, you know, depend on them, so we would like to drive some effort here with the upstream kubernetes community to make the kubernetes control plane more scalable.

A

So, but you know this has to be a community effort, and you know people from other, maybe open source communities or other vendors joining these efforts would be really appreciated.

B

Will you be here for the rest of the week uh yeah if those folks want to get in touch pretty easy to find, let's give another round of applause for outfit excellent talk, yeah. Thank you very much.