Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Going from 5s to 5ms: Benefits of a Node-Local DNSCache - Pavithra Ramesh & Blake Barnett

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Going from 5s to 5ms: Benefits of a Node-Local DNSCache - Pavithra Ramesh, Google & Blake Barnett, Postmates

DNS is one of the most heavily used services in Kubernetes clusters.The kubernetes community has long struggled with these mysterious 5s connection delays or DNS failures. This talk discusses how this problem is (mostly!) solved with the node-local DNSCache, a feature introduced in Kubernetes 1.13. We’ll start by diving into the DNS problem, the motivation for the feature as well as the design and implementation. Attendees will learn how to use this feature on their clusters. In addition to the latency issue itself, visibility into DNS has been a pain point for many users. Not any more! The metrics exposed by the local DNSCache provides insight into user workloads' DNS needs. We’ll show some of these metrics, along with a demo of how the feature works. Finally, we will share the performance improvements observed in a large-scale production deployment running node-local DNS Cache.

https://sched.co/MPbO

A

B

Thank you so much for taking the time to attend this talk. It's benefits of no local DNS, cache I know it's the last session for the day, so we'll try to keep it short and fun and make sure you're all ready for the evening party awesome like like it was just introduced and pavitra Ramesh I've been working on kubernetes for over a year now, mostly in the DNS space plate and Before, we jump into the content I'd like to get a sense of how many of you here have already used. No local DNS cache awesome.

B

Ok, thank you! Thank you for trying it out and how many of you have had to debug DNS latency in your clusters. Ok, I, expected it to be a superset but Wow. Ok, that was more than I expected cool, not cool that you had a debug but good to know. That is a common problem and, finally, how many of you are wondering if you're in the wrong talk I'm just kidding awesome.

B

Thank you for not raising your hand, so everybody here wants to be in this talk. That's great! So on that note, let's jump in the agenda for today is the following: we're gonna start by giving a short intro to what no local DNS cache is and why we needed to answer the why we are going to quickly look at what the current DNS therapist in kubernetes and how local fits in then we'll talk about what new metrics are available to you.

B

When you enable this feature, then we have a very interesting section that Blake will be covering which goes over what what real-life improvements in an actual production cluster heart after enabling the feature, then we talk about. Ok. How can I deploy this in my cluster? Ok in any of your clusters that is, and then future work what's coming and finally, we will have time for questions as you can see, it's a packed agenda, but we will keep it brief and hopefully fun.

B

Ok, let's jump right. What is no local DNS cache, an add-on that runs a DNS caching agent on every node, so there's one instance running on every node as a daemon set and all pods that run on a specific node talk to their local instance for all the DNS needs. So that's the short definition of what that feature is, and why do we need this for this I added a screenshot of some of the github issues, tagged yourself if you participated in one of these.

B

These are issues asking for a feature similar to this or it's discussing DNS latency and the conversation converges to a solution like what we built. These are just some of them. There will be more and many of them closed now, since we got this feature out, but why are DNS latency issues showing up in kubernetes clusters? These are just some of the reasons I put out there. Not all clusters are the same. So if you have experience it could be one of these. It could be a different reason.

B

It could be a mix of some of these, and even these reasons are not independent. They are intertwined. So the first one I'd like to talk about is too many queries all at once. So too many parallel queries and by v4 and v6 I actually mean a and quad a records. So why does this happen? Think about it?

B

If a cluster or a client part is trying to look up foo.com, the client is trying to issue only one request, maybe to request a record and a quarter a record, but this is actually translated into several more requests and you can see all the search path suffixes that get had it in that red box there. So all these suffixes are added to make it five queries of type a and five queries of quality records.

B

So you can get ten or even more queries for every actual query that a pod requests and this can increase the chance of netfilter race, condition and I'll link this here, there's a very nice blog post. That explains why this happens when there is a network address translation, so you can take a look. I say it is pretty hard to pinpoint that this is what you're running into.

B

If there is a packet drop, the metrics to track- if this is happening, aren't super clear, but it might be one of the reasons, but parallel queries are problematic. Even otherwise. Here are some of the other reasons too many DNS queries, overwhelming connection tracking tables and parallel queries can make that problem worse too, for every connection that that happens on your Lord, whether it's Andres translated or not, there is a tracking going on. So each of these connections occupy an entry in your connection tracking table which has a limit.

B

So if you have more connections opening up, then there is space on the table. Those are gonna, get dropped and packet drops are not great, not good at all, especially for UDP. What are the other reasons again? More parallel queries means DNS mask concurrent connections limit. This is less of an issue now that core DNS has become the default, but if you're running cube DNS, you might still run into this. The default is 150 connections.

B

Concurrently to a single DNS mask instance, there could be cloud provider limits that allow you to do only a certain number of queries from a node and, if you're doing more than those those might be dropped as well, causing the same latency problem. And finally, this is actually a reason rather than a separate point, since UDP is a an unreliable protocol. If packets are dropped somewhere, the client doesn't know whether the packet is really dropped or coming back really slow. So it's gonna give the benefit of doubt and wait the entire time out.

B

So that's by default five seconds. So these are some of the reasons that we might be seeing: latency DNS latency in kubernetes clusters. Now, even without node local cache, there are a bunch of solutions or workarounds that you can. That can help in this case. In these cases, and let's look at those, the first one is using an option called single request, reopen what this option does. Is it tells the resolver to issue sequential queries rather than all at once and by doing this you're ready.

B

The number of parallel queries do note that not all DNS resolver libraries support this option. The muscle implementation that Alpine uses doesn't support this specifically, but that this is one way to reduce the parallelism in the queries. Then there are other options that reduce the number of queries itself. Reducing your n dots in dots is another DNS option that you can set to a lower value. This basically disables the search path, expansion that we talked about earlier.

B

So it says, if I have certain number of dots in my query already, don't even bother trying to expand it into ten queries. Just give me what I asked for just look. Look up what I asked for running cube, DNS, DNS, maskers daemon sets to spread the load. That's another solution. You can try again.

B

These are some more workarounds at the time and modifying DNS mash parameters to increase the number of connection limit and then using a more reliable protocol using TCP for your DNS connections, with the use vc option again, I haven't had too much luck with this being consistent. So again, it might work depending on what client images you use and another one I want to point out. It's relatively new I know how to path plug in in core DNS. This basically still gives you the search path, expansion feature, but it moves that logic to the server.

B

So the client still requests only a small, only a single or two queries, but all the search expansion happens in the server. So you reduce the parallelism again. So these are some of the workaround. So, let's see how local DNS fits in here and here I have a diagram of how the workflow would look like and where the cache fits in I can mention. You have the gray box, which is a node, and we have a caching agent that runs one per load so yeah within the blue box. Here, that's the local DNS cache.

B

It gets its own interface and I know it's exclusive IP address a link, local IP that it listens on, and requests coming to, that IP will be handle by the local cash. When you enable this add-on, you also the cubelet flag to change the cluster. Dns is automatically done for you, so that's the flag that tells cubelet to write the name server in the pods, so it will now write this link. Local IP in the pods resolve account, so pods will talk to the node local cache, rather than cube DNS service.

B

Now, let's see what happens when the client tries to look up some some hostname. If you look at the lines in green, it can send a request via TCP or UDP. Udp is by default. It hits the local cache and if there is a cache hit, the response goes back to the client quad directly and I want to point out that connections between the local cache and the client pods are not tracked, meaning they do not occupy entries in the connection tracking table. So you bypass the net filter layer for these connections.

B

So if you have a cache hit, you don't have to touch the network address translation code at all called paths at all. However, if there is a cache miss which is the arrow in the bottom, you will go to no local cache store, but no local cache.

B

We will talk to the chip, DNS service IP, to get the authoritative answer back, and this does require address translation, but it uses TCP rather than UDP, which is the more reliable protocol, and we also reuse connections so you're not going to create one connection entry for every request that the pod made that wasn't in the cache. So this is how local fitting let's see how this improves the problems we listed earlier. So now the too many queries problem is fixed to many parallel queries. Problem is fixed because you're not doing connection tracking.

B

So the the tools queries are not filling up the tracking table, they are also being cached, so they are not going out to cube DNS all the time. There is no net filter address translation code, so the anything could potentially cause there is this out of the picture. Nearness mask concurrent connections limit same thing, reduce number of queries. External queries go directly to the node resolver. They don't even have to go to class, ready, NS and piggybacking.

B

On that same point, since external queries go directly to the node resolver from that node you're able to better utilize any node specific limits there might be for DNS queries.

B

And finally, we talked about UDP, so we are upgrading the connections to TCP, so that's taken care of as well. These are some of the performance benefits you get by enabling node local DNS cache, but that's not all you get extra visibility and metrics as well.

B

No local DNS uses core DNS as a cache, so it uses a cache plug-in the forward plug-in and few other plugins from Co DNS. So all the metrics that Cody and as plugins export are available and the best part is they're available on a per node basis. So you can get insight into what your DNS request patterns look like on a personal basis, and maybe you can pinpoint that to certain pods on your node as well, so you get that extra granular inside here are just some of the stats I put out there.

B

This is just the tip of the iceberg. There are several more in the cache plugin you see. Cache hits misses in the forward plug-in, which sends the request to the upstream. You can see what the request counter is what their latency distribution looks like. You can also see this on a per zone basis, so you get separate stats for your cluster domain. You get separate stats for external queries, which is nice to see the distribution of latency, particularly okay.

B

Hopefully this all sounds good in theory, but does it actually work and to cover that I would like to invite Blake to talk more.

C

So a post mates we have been running kubernetes for a little over two and a half years, and we've hit all of the that were listed around DNS.

C

We initially tried to mitigate this problem by deploying cute DNS as a daemon set. It works fine. It does require modifying Kubla tarts to point the local node at its at the resolver, but the problem with it is if you're running just cube DNS as a daemon set. All of those daemon set pods are watching all pods via the API server and over time it becomes pretty resource intensive. So we wanted a little more streamlined solution to this and we were very excited when I found out about the development for this project.

C

So we started testing it right away.

C

Just some stress testing scenarios that we did we most of them target a single cluster domain service, the community's default service, so that we eliminate any you know additional variables in the system uses confirm DNS, which is written by Justin Santa Barbara, the, if you don't know, he's instigated a BOS and does cops as them he's the main maintainer on cops. He he was nice enough to provide us this code and we did a whole bunch of different scenarios testing this.

C

So his each of the confirmed, eNOS pods, send about 200 queries per second 100 of a records 100 of quad-a records, and so most of our tests deployed about 240 pods on fairly small clusters, just to get a controlled environment. To stress these out. We did this on various providers and it was all pretty similar on all of them. So this shows how easy it is to trigger the contract overflow, especially on smaller node types like a default, decaying instance type or something any anything on any managed provider, usually they're, quite small.

C

To begin with, and the difference with no vocal is pretty dramatic. You can see. All of these are five-second timeouts, I'm filtering for anything above 100 milliseconds, and there are a lot of five-second timeouts.

C

This is the same cluster with no political cash deployed. You can see there's a few minor outliers, but by far most most queries are under 5 milliseconds.

C

This is a similar test. This one used a range of DNS names that triggered search paths. Sir sorry search path, expansion and a bunch of external queries. This results in hitting the contract limit very quickly because of the explosion of queries from all those different types, and you can see on the right side there. After we deployed the local DNS, it's really improved latency.

C

This is a similar test, just a different view of the distribution of latency, and you can see none. There are no queries over 16, milliseconds and 60. Milliseconds is an extreme outlier. It's also important to note that on much larger nodes, you probably won't hit contract, but we do still see a lot of five-second timeouts from either in that filter. Race conditions need to be packet, loss, there's a range of possible problems here and we sidestep almost all them. Using this.

C

So we deployed with local cache. It was great. We solved almost all of the problems with the cluster queries. We did notice, however, that external queries were a little bit erratic.

C

A ton of cash misses because the queries were not happening happening quickly enough or just a range of external queries, so they weren't getting a lot of cash hits. We made use of the create the prefetch plugin for Cortinas, which is quite effective at reducing the overall latency there. You can see the dramatic change once we enabled that option, and if you have questions about that, I can answer that after Wow, this shows our production query: the production cluster.

C

We have around 50,000 queries per second at peak and the nodes are really large. There I believe 64 core instances, and we have some very large workloads running on them with extremely high throughput.

C

The core DNS is the cluster DNS service, and you can see that all of the latency there is sub millisecond, which is really a huge improvement over our previous set up, and you can only see the top four notes there, but there's I believe around 120 nodes in that cluster. All very large nodes.

C

And a back to Pavitra, okay,.

B

Thank you so much Nick I want to point out that Blake was one of the first users of our feature and he's given great feedback from the time we released it and I'm so glad we could present it at a wider audience today. So yeah, hopefully now you're excited or interested in trying this feature out in your clusters. So how can you run it? So here are some of the steps you can follow to run it.

B

If you are using the e2e scripts to bring up your cluster, you can use the command up there in the first option. There it works for 1.13 and above clusters. You can run it that way, but on any environment on any cluster, you can use the Yama. The links here to do just a cube, cuddle apply that lower link is the newer yama and you can.

B

You can try the one, but you wouldn't need to do the cluster dns change its a flag to cubelet, which you will need to modify so that pods automatically start using your local cache. But if you want to keep the current setup and cannot change cluster DNS flag for any reason, you can still use it on an existing cluster and just in Santa Barbara I and wrote some scripts that you can use to make this happen, so do check out the link and the pull requests there or you can reach out to us.

B

If you have more questions on how to run this, this is all of the ways you can try out the feature, and it would be great if you can provide your feedback after you try to so now. The only section we have is what's coming in the future. We release the feature in alpha in 1.13, so we are graduating it to beta in 115 and yeah we did like I mentioned.

B

We got great feedback from some of our users, so it would be great to get more and the two items that we are working on or proposals are in progress, are each a and auto path. So the one question that I got after folks tried out. The feature is like hey: what about high availability for my DNS? Now? No local DNS introduces a single point of failure. What, if it's down? What do I do so? I, don't want to start by saying that this is a common problem for any node specific agent. I take Kuebler.

B

Take your proxy. It's one per node with the same H a or non H a story. So to some extent you could you consider your node. As you know, failure domain so anything fails its contained within that domain. Building that node, however, I understand there can be scenarios where DNS downtime cannot be tolerated, and for those we have a few options. One of them is to run to daemon sets instead of one and use the same IP address. So whichever one is up can answer queries.

B

We have another option that I would go into some detail about, and this is what this will. Let you keep the exit seeing single diamond set approach, but we can do some iptables drinks to provide a chain. So let's take a quick look on how this works. I have the link to the cap here, which has a full detail. I'm gonna go through it a little bit fast, so the picture is still similar to what we had seen before. You still have the node.

B

You have the per node cache, but if you notice it listens on the link, local IP, it listens on an additional service IP and not just any service IP, but it's the cube, DNS service IP. So what this lets you do is the client pods can talk to node local cache without changing anything on their side. So you don't have to change their cluster DNS, it's already pointing to the dart, and so they hit the cluster.

B

Do they hit the node local and we will install custom IP tables rules so that if no local DNS is up and running, queries will go to no nor local. If it is not, then those custom rules will be removed which will result in the normal behavior, which is the service IP and then going to the cube DNS pods behind.

B

So how do we decide that? So the only other thing is then: what happens when no local needs to talk to cube DNS on a cache miss? What IP does it use so we create a second service with the same selectors has cube DNS service, so it is backed by the same cube, DNS pods. This is this new service. Ip is what no local will talk to when it needs to fetch entries for a cache, miss.

B

So the one question here who decides when to add the custom rules to route tune or local when to remove them and send it to cube DNS. That part is a little bit of a TBD, but you can put this logic in the same part as a sidecar container, or you can put it in any other demon set that is running host networking.

B

That's already doing a bunch of iptables manipulation, maybe, but that would essentially have to ping the link local IP to see if no local is alive and if it is keep the rules, if it is not remove the rules, so the packets go to the cube DNS service, which is the natural fallback we want anyway.

B

One caveat is that this approach wouldn't work for IP vs mode of cube proxy because of the way Q. The IP vs called it, creates its own interface and binds the IP. So we cannot reuse that trick here, but this is certainly an option you can look into if H a is a concern and I have the link there for the more details they that it's captured in the Kappa.

B

The one other feature we are working on is auto path and I want to take a few minutes to talk about this. We discussed what out of path is. It is the idea of moving the search path, expansion, logic from the client to the server, so why do we need Auto path or why we need these search. Paths is because you want to be able to discover other services using their short name say you created a my new service in the default namespace. You would like to resolve it by just saying my new service.

B

Instead of my new service default or service or cluster or local, you want to use the short names or you want to use named or namespace to find these service IP addresses. So that's what the search paths, let you do so, if you remove it, you're losing that ability so with Auto path. You get to keep that ability, but you get to reduce the number of parallel queries on the client side as well. So yeah I went through most of the benefits here.

B

The other benefits of this proposal are that you can extend it to any domain that has search paths even outside of kubernetes. It can be applied anywhere and also I want to point out that core DNS, auto path plug-in already supports this today. But the proposal we are Pro we are putting forth is less resource-intensive and the details you can find there, but I will go through quickly. What the new workflow will look like with this.

B

Let's say we introduce a new mod and all these names are TBD it's in progress so, but we will introduce a new mod say cluster first with Auto path and in this mode, cubelet will create a single custom search paths instead of the fibers for whatever search parts it creates, it will be just a single one and that will have the namespace and the suffix encoder in it, because, depending on the part, you're running on your search path is going to be different.

B

So this is the search path that will be in the pods result or column. Now what happens when the pod looks up an IP or looks up a domain name?

B

Let us take a look now when a client pod looks up a hostname, say foo.com, the search bar expansion kicks in and that's the domain that is going to be sent to no local, no local cache we'll have the ability to recognize this custom pattern and extract namespace and cluster suffix from it, and once it has both these variables, it knows to construct what does our new list of search paths need to be, and once it constructs this, it can put it as a needy and a zero option.

B

It's an extended dns value in the in the request packet key value, so we are going to introduce a new option number and we're going to put the list of search paths as the value and if the cluster dns knows to recognize this key value option, it can look at the search paths and fan out into multiple queries. So that's that's what the last green box is doing there. It gets the cluster all of the search path from the node local in the packet just needs to look at it.

B

Append all of them send out the queries return. Anything that had a non-empty answer, so that is how this would work and you don't need to watch any resources in order to figure out what your such paths need to be and you've moved the search path, logic outside of the client, so this is one other thing. That's in progress like I said feel free to comment on the cab feedback would be great. I believe this is the end of what we had here.

B

So I hope this was useful and got you excited to try it out on your cluster and I have the test that we talked about I. Have it running on my cluster? If you want to take a look after I'll be around for questions, we'll both be around for questions, but I have the set up. Also, you wanna take a look. Now we have time for some questions.

D

Thank you for the presentation. My question is regarding the details for the DNS cache and.

C

D

That is quite long. Then we could get. There is a risk that the application could get like an old version for the DNS, and that would impact some timeouts on the request. What do you think? How do you think we could go around that are.

B

You talking about timeouts for an X domain exam.

D

Of your application like what what call the local DNS cache, it would get like an old version for another hat right if it tries to reach it. It's it's not there after, for example, the company, this new deployment or something okay.

B

Yeah yeah, which so we do set the timeout to a low value for for the negative domains, so that, if it's a case that, if a client looked it up and we cashed it as no no host available, but the entry actually came up, you don't want to run into the case where that's not reflected so for negative answers. We do set that value to a low number, but I'm, not sure. If that's answering the question.

D

Kind of like we said it like yeah,.

B

We said you'd do a low number for a scenario which might be something out of what you described. That is part of the Yammer. We set it to a low value.

E

Hello first, thank you very much. I still keep really bad memories of was long weeks, trying to debug DNS latency in my clusters, and it's really nice to see a technical implementation of a solution for that. So thank you. The second thing is a question.

E

We we do have a lot of latency sensitive applications in our clusters, and one of the challenge we see is also the I think you haven't I think you have an answer for this, but one of the challenge we see is that DNS resolution for most HTTP libraries is on the critical path and the TTL for community services is five seconds. So every five seconds you need to actually make the full DNS the cache will expire.

E

It will be a cache miss and you need to do the DNS resolution and please take some time- and you do this every five seconds for an IP but barely changes. Is there a way to do some prefetching behind the scene or something like this? You mentioned prefetch before so I'm, really hoping about the solution. So yeah, that's my question. Okay,.

B

Do you want to present.

C

So yeah the prefetching will work for any of the you can set it up for clustered domains also, but by default it is at least the way I showed it. We set it up only for external queries, because that's where it metals matters the most for us, but you could configure it any way. You'd like with you, you can tweak the settings. We have it set quite liberally so that we always get the most popular queries.

C

Pre fetched, you can have it a little less aggressive and have it so that your applications have kind of a specific configuration. There's. Also the the cache settings and coordinates are quite flexible. You can set different cache for different subdomains. If you wanted to break it out that way, and then you could also have different default cache timeouts for for those different subdomains.

A

Hello and thank you again for the in very nice talk about the subject. My question is about Auto. Pass current of limitation always requires that you have both verification enabled. It sounds like your new approach may not have to do with that anymore, because, and it is related with a footprint of coordinates in the system. So can you maybe tell myself it's more about it? How basically only need to be enabling, from now on.

B

Yeah I'm not I'll, ask the question again because I'm not sure I got it I'll ask you can tell me if I got it right, are you asking how the new Auto path compares to what the existing one in Karina's is? Yes,.

A

Especially regarding footprint of core DNS pods, meaning that the verification the pod verified setting is always memory. No it's hogging memory and seems that now there can't be a way out of it because of how you implement the other path into the local right.

B

So the auto path as it works today in Korea Nestle.

B

Yes, there needs to be a watch, that's happening in order to map the pods IP to the pods namespace, so it has to watch the pod resource in order to do this, but with the current approach, we don't need to do that, because we've pushed it to the client to encode this information and sell it out so again, I think it's hard to pinpoint how much of the memory usage is through auto path unless you've had some benchmarking, but that Biff that you might be seeing shouldn't be there with this other approach, because the only thing would be that it'll still fan out into multiple queries and collect the answers, but there shouldn't be something that's growing, as there are more endpoints in your service.

B

For instance. Thank you, hello,.

F

Regarding the contract issue in the kernel, is there any issue that this be solved that point some point in the future? It's just too hard to fix.

B

Running into the limit, like too many entries.

F

There is a race condition in the colonel there any hope that this can be fixed. The.

B

Link to the blog and yeah I, don't know as much as the authors of the blog, but they did submit three patches I believe to the colonel, and two of them have been accepted in different kernel versions, so fixes are definitely on the way for that. I did talk to them at the booth yesterday, so affixes are definitely under way for that.

B

A

B

Okay, thank you so much and enjoy the party everyone. Please.