Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nondestructive Forensics: Debugging K8s Services Without Disturbing State - Alex Leong, Buoyant

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Nondestructive Forensics: Debugging K8s Services Without Disturbing State - Alex Leong, Buoyant

If debugging distributed systems is like solving a murder mystery, then we need a way to gather clues without disturbing the scene of the crime. Some application errors (resource leaks and race conditions in particular) can take a long time to reproduce; other errors might occur unexpectedly in production. When this happens, we want to gather as much information as possible without restarting the pod. Tools like detailed logging, debug sidecars, and Linkerd's "tap" are only usable if we had the foresight to include them before the error occurred. Ephemeral containers promise to be a useful tool, but are currently in alpha. We will compare these techniques and then see how we can use direct access to the node to enter a pod's network namespace and do detailed network debugging without needing to restart any running containers.

https://sched.co/Zeqs

A

Hi, this is non-destructive forensics, debugging kubernetes services without disturbing state. um If you have any questions throughout this presentation, go ahead and type them in the box below and we'll do a q a at the end.

A

uh So I want to start with this kind of ubiquitous tweet, uh which says we replaced our monolith with microservices, so that every outage could be more like a murder mystery from honest update. I really like this quote. I think it is very apt because when you're debugging outages, you know you have to do a lot of the same things as in a murder, mystery. You have to collect clues and evidence. You have to consider the suspects and you have to kind of mentally reconstruct the series of events that you think happened.

A

So my name is alex. I work at a company called buoyant and I am a maintainer of the lingerity project. I also really like board games and I really like debugging microservice murder.

A

Mysteries, uh so to give you a little bit of context, um I wanted to talk about what link rd is so link rd is an open source service mesh, it's part of the cncf as an incubating project.

A

It's got open governance, of course, because it's part of cncf and it's ultralight, security, first written go and rest, and if you're not familiar with what a service mesh is, the basic idea is that we run a sidecar proxy container inside each one of your pods, and this sidecar proxy is responsible for intercepting all traffic that goes in or out of the pod and proxying it along and then also doing some kind of interesting stuff with it.

A

So that includes recording metrics about that traffic, uh doing intelligent load, balancing um and so on and so forth. uh Adding security, mtls automatically and what's kind of cool about this is that it's totally transparent to the application. The application doesn't know that this is happening, it's all kind of layered on.

A

um But the thing is that, since these proxies are handling all the traffic going through the network, it's really really important that they don't have any bugs, or at least that they don't have any bugs that's going to interfere in any way with the traffic they're proxying.

A

So one of the most interesting things that I get to do is I get to investigate bug reports um and this is kind of fun because it means I get to do these kind of investigations and and do this kind of murder, mystery thing and try and figure out what went wrong.

A

uh So I want to start just by talking about what what is this detective work and what does it entail and why is it difficult um and then go through some of the tools that I use to do these investigations? So I wanted to talk about a feature linker which lets us tap into.

A

A

Into a uh node in your cluster to get an even better look at what's going on, and in order to talk about that, we kind of have to talk about what what is a pod uh anyway and and what happens when we kind of peel back of the layers of abstraction look at what's really going on um and then. Finally, I want to talk about a feature. That's coming up in kubernetes, where this alpha right now called ephemeral. Containers.

A

So when you get a bug report, uh probably the first thing you're going to want to do is you're going to want to reproduce it. If you can reproduce that bug it's going to be much easier to solve than if you can't, but sometimes this is really difficult. Some bugs are very, very difficult to reproduce because they may only happen in certain environments. They might only happen at certain scales or at certain load levels or after a certain.

A

You know, amount of time has elapsed or when some other you know set of conditions has been met, and you don't necessarily know what those conditions are.

A

So this can be very frustrating, especially when you get a bug report that says you know this thing happens, but I can't tell you when- or I can't tell you why, uh and you have to kind of figure it out on your own. So sometimes the best you can do really is to replicate uh the you know.

A

What's in the bug report, as as closely as you can replicate that environment to the best of your ability um and then just let it run, and you know, keep your eye on it and hope that you trigger the bug and if you're lucky, then eventually the bug will happen.

A

And so when, when this happens- and you have triggered this bug that you've been trying so hard to reproduce, you want to gather absolutely as much data as you possibly can, and you don't want to kind of squander that state that you've gotten yourself into so you want to collect. You know logs. You want to collect metrics, you want to collect. You know everything you possibly can about. That's about that state you're in you want to look at the network state, the state of the sockets. You want to do a traffic capture.

A

If you can get a traffic capture, you want to get kind of as much data as as possible, and you want to avoid disturbing that state because then you're going to basically have to start over.

A

If you end up restarting the process or restarting the pod or interrupting any of those connections, you may kind of lose that that state you're in you have to start over and reproduce the bug come from scratch, and if for these bugs that are very difficult to reproduce, if you don't have a reliable way of doing that, that can be very, very painful and time consuming.

A

So, in other words, you want to collect data without disturbing the scene of the crime, and so I call this non-destructive.

A

A

um And for this talk I want to really focus on bugs that have to do with service to service communication, and the primary reason for this is because this is where I have the most experience. You know I work on linker d liquidity proxies network traffic, so almost all of the bugs that I investigate have to do with communication between services and some at some level, um but I think it's more than that. I think communication between services is really what is essential to microservices.

A

You know, if you don't have services communicating or if there's nothing, going wrong with that communication, then it's really kind of no more interesting than than single services or monoliths. So I think, looking at service service communication really captures something that's essential about microservices.

A

Another reason is, I think these are just really hard to debug. We don't necessarily have a lot of tools, or at least we're not necessarily used to using the tools for debugging service to service communication, and you know it's just more complex. You have many different moving pieces, they're all running concurrently, so you have a high degree of parallelism. There's a lot of opportunity for race conditions. Network connections are uh not totally reliable, so there's failures that can happen not only within the services but also between them.

A

So there's just a lot going on a lot that can go wrong and so debugging these types of problems can be really difficult.

A

And, finally, the other reason why I think it's really interesting to look at service to service communication is that it kind of allows us to treat each service as a black box.

A

Just looking at its communication, we kind of see both its inputs and its outputs and and that lets us help or that helps us narrow down our list of suspects and kind of figure out which service is behaving and which is not without needing to look inside it, and I think that's a really powerful technique for for kind of narrowing down the scope of our.

A

A

So I think about every service as having a contract, in other words a set of behaviors that it needs to uh that it needs to do or a set of rules that it needs to obey, and sometimes this is explicit.

A

You might have some formalized api specification that exactly spells out what your service is supposed to do, but more commonly, I think, it's kind of implicit as someone who has designed this system, you just have it in your mind, what the correct behavior for each service is and how they're supposed to coordinate and work together and get along and- uh and so in other words, you know what constitutes correct behavior and the idea behind microservices is that you have all these services each with their own individual contracts and, if they're, all doing the right thing, then the service, the service architecture as a whole does the right thing, and if that means that, if the service architecture as a whole is not working, that means some individual, or at least one individual service is not satisfying its contract.

A

It's doing something wrong and, like I alluded to earlier, we can look at the communication between services in order to kind of identify uh which service is not fulfilling their contract. You know treating them as a black box, which is not doing the right thing, and it's also worth noting that these contracts exist at multiple levels. So you may have application level contracts which say things like.

A

If my service gets a request for a certain type of record, then it's going to return it or if I get a new user creation request, then I'm going to write a row into the user's database, but there's also network level contracts which say things like. If I get a tcp packet, I need to acknowledge it, and so, in order for your service to behave correctly, it has to obey its contracts. Kind of at every level.

A

uh So the first feature I wanted to talk about, or the first tool uh in your toolkit or in my toolkit- uh is called linkery tab, uh and this is a feature of linkedin that lets you tap into http streams. So remember, I was describing linkedin earlier and I said that it runs a sidecar proxy inside every one of your pods that intercepts all of the in-going and outgoing traffic.

A

One kind of powerful thing you get from this is that you can now kind of connect to any proxy and say: hey, tell me about the traffic that that's flowing through you. You know describe it to me. uh Let me tap into it as if I was like doing a wiretap, and this is really powerful, because you know it lets us see this data in transit.

A

It lets us see it in real time as it's moving, and so we can look at requests and responses as they flow through the system in real time and authorization for this is controlled with our back. I think that's really important. um That means that you can set up our back rules that specify you know which users or which accounts are allowed to tap into which resources you don't have just you know, cart, cart, launch access.

A

uh So here's what that kind of looks like using the linker dcli. You can say something like liquidity tap. You can give it a namespace and a resource, and it will just spit out a stream of events that describe the requests and the responses that that proxy is seeing.

A

So in this case, I'm just showing three events that that showed up right away. We've got a request, a response and an end. So the end means that the response has has finished has completed, and you can see that there's just a ton of really cool metadata in here. So, for example, we can see the source of that request. We can know the ip import that it came from and also the destination where it's going to.

A

uh We can see other http metadata like the path here. So in this case this request looks based on its path. Like it's a grpc request, uh we can see stuff in the response. There's the status code and the latency, and then we can see trailers uh in when the response has completed. So this was a grpc uh status. Okay, trailer. We can see how long it was streaming for and how many bytes were in the response, so tons of really really uh cool metadata.

A

um Oh and the other thing that I wanted to say about this- is that uh you can. Of course, this is maybe a little difficult for humans to to consume, but you can also pipe this into other text processing tools to make it uh to kind of filter down to what you want to see. So you can pipe this into grappler awk or do whatever you want, um and it also supports json format. If you wanted to treat this a little more programmatically.

A

But since it's a little difficult for humans just to eyeball to see with their own eyeballs, this is also available in the linkerity dashboard. So here's kind of that same data, but but in the the web dashboard- um and you can see here- we've rolled things up by their path, so everything all those requests that have the same path uh get rolled up together until we can get account of how many requests there were and we can also calculate success rate based on the http response status code.

A

So that's a really cool way of getting a view of that data at a glance, and this is really powerful for for seeing which requests are happening between which services and how long are they taking and are they successful and are they or are they not successful?

A

And you just get a lot of high level data about that communication very, very quickly and very very easily. um So, like I said, this is like super super convenient. You know it's one cli command and you just get all this data, basically for free um and it's very rich in the amount of metadata you get, and it has this very nice property that we care very much about that is non-destructive. I didn't have to restart anything I didn't have to you know, add log lines to any program and recompile them.

A

I was just able to directly tap into that existing http stream.

A

Now the downside here is that, of course, this requires that you already have linguity running. So, if you don't already have linker d installed in your cluster and in your pods, then it's not non-destructive anymore. You would have to restart those pods to add the proxy, so it's kind of only non-destructive if you already have it there.

A

The other thing is that we were only able to see metadata. We weren't able to see what was actually in those request or response bodies. That's coming in a future release of liquidity. It's on the roadmap, it's something that I'm very, very excited about because being able to see exactly what was in those messages in those requests and responses. I think would just take this to the next level. You'd be able to do a lot more uh debugging.

A

If you were able to see you know exactly what are the contents of these requests and responses, and are they correct? So that's something I'm really really excited about, um and the other big big drawback here is that this is http. Only grpc is also hdb, uh and that kind of means two things you know. First of all, it means that if you have traffic, that is some other protocol, like um you know my sequel or redis, or something else.

A

um This isn't going to work, you're not going to be able to use this tool, but I think the other thing that this means is that, even if your traffic is http you're only getting http level data about it, you know in those events you're not seeing anything about individual frames, you're, not seeing anything about individual connections or individual packets, um it's all at the http layer.

A

So if your problem that you're debugging uh goes below there and is more complicated and you need to dig a little deeper, that's kind of where tap stops and you can't get anything lower level out of it.

A

So tap is always kind of my first line of defense is always the first tool I turn to because it's just so convenient and gives you so much data so quickly. But often I find I need to dig a little deeper and kind of drill down to the next to the next level, um and so that brings us to debug sidecars.

A

So it turns out. There already exists a lot of really really powerful tools for doing network debugging. um Some of my favorites I've listed here, netstat and ss for looking at the state of sockets on a system, tcp, dump, wireshark and t-shark for grabbing, live captures of traffic, digg for doing dns, queries and then curl and nghttp for doing http queries.

A

And these are super super powerful tools, um but they might not exist in the container that you're debugging. So, ideally, what you would want to do is you would want to use exact to shell into a running container and use these tools to see what's going on, but those tools might not exist in the container. In fact, your container might not even have a shell, so this is where you can use. What's called a debug sidecar and a debug sidecar is just another container that has all of these tools that you would want.

A

It's got a shell. It's got all these powerful networking tools installed into it, and you run that debug container inside the pod, the same pod as the container you want to debug and because they're both running in the same pod, they're going to share a network namespace, and what that means is that you know they'll both share an ip address, they'll be able to talk to each other on localhost and critically, they'll be able to see each other's network traffic as if they were on the same host.

A

On the same, you know, network.

A

So this is a diagram of kind of what that looks like. So here we have one pod. It's called the application container, which has got all these connections to it. That's doing whatever it's doing that we want to debug, and then we have the debug container, which is a separate container running in the same pod and that debug container's got all of the the fancy network tools that we want and we can use coop control exec to run a shell inside that debug container and then use all those tools to to see.

A

What's going on in the application container.

A

um And so linkerity has a command that makes this really easy. uh We can do liquidity inject and we can use the enable debug sidecar flag and what that will do is it will just add a sidecar container to to whatever workload you specify that has all of these. You know debugging tools um and then it's just a matter of exacting uh here's coop control exec into that debug container, which gets you a shell into that container, and then you can run tcp dump or netstat or whatever tools and kind of debug away.

A

uh So this is really great. I use this a lot because you get basically any low-level network debugging tool you could possibly want. You know anything you can put in a debug container. You can get um the other nice thing about this. Is it doesn't require linkerity?

A

I just show you a command, a linkerity command that lets you very easily add a debug sidecar container, but this is not necessary. You could just add that sidecar container to the manifest manually, you could even roll your own debug sidecar container. That has you know whatever tools uh you want. uh So that's totally up to you.

A

um The big downside here is that uh it's it's destructive. You can't add a container to a running pod. You have to restart the pod in order to add a new container to it, and so this violates kind of our big thing that we wanted. We can't we can't we have to disturb the crime scene in order to add this debug container, so this is kind of not going to necessarily work for us.

A

So this brings us to direct node access. So if we want to use all these powerful debugging tools, but we don't want to restart the pod well, what if we just ssh directly onto the node in kubernetes, that's running the pod.

A

So here's that diagram I had before we've got the application container and the debug container running inside the pod. But we know pods don't exist in a vacuum. They are run on nodes. So here we have a node with three pods on it, um but this diagram isn't totally accurate either because pods don't actually exist.

A

um Here's what it might kind of more realistically look like you've got the node and then you've got the container runtime on that node and it's running a bunch of containers- and you see there's no reference to pods anywhere here in this picture and that's because a pod basically is just a network namespace, uh at least for our purposes. So you'll see that some of the containers are different colors.

A

That means that they're running in different network name spaces that correspond to their pods, and you know, like I said earlier, a network namespace basically just means that all of those containers share an ip address. They can talk to each other on localhost and they can kind of see each other's traffic as if that was one network entity, and so, if we just ssh onto the node, we're not going to be in the right network namespace to debug.

A

One of these containers we're just going to be kind of at the node level, and what we really want is some way to enter into the network namespace of the pod that we care about in order to to debug it, and it turns out there's a command that does exactly that. It's called ns enter and it lets us execute commands in a given network namespace.

A

So all you have to do is give it a process id of a process that you want to debug and then it lets you execute a command in the network name space of that process.

A

So here's how I would typically use that this is me after sshing into a gke node in my gke cluster. I start running docker ps, which lists all the containers that are running on that node. I look for the one that I want and I note its container id then I can use a command called docker inspect to get the process id of that container and then I can use ns enter to run whatever command. I want uh in the network namespace of that container.

A

So in this case I run tcp dump and I get you know a traffic dump of what's going on there.

A

uh So this is really really cool. um We get all of our low-level tools. We can use tcb dump or netstat or whatever we want it's totally non-destructive. I didn't have to restart anything or interrupt anything to do it, um but it did require sshing directly onto the node, and so this is something you may or may not have access to, depending on how your kubernetes cluster is administered, um and it is a little bit you know uh complex, I had to ssh onto the node.

A

I had to get a pid I had to you know, figure out the network name space and kind of jump through these hoops. In order to do it.

A

So the last technique I want to talk about is ephemeral, containers and the this is a new feature. uh It was introduced in alpha and kubernetes 116., um and this is basically all all we ever wanted. This lets us run add a container to a running pod without restarting it. This is kind of what we've been hoping for all along. This is the best of all worlds. We get all of those powerful low-level tools, it's easy to use. It's non-destructive.

A

All we have to do is take that debug, sidecar and just add it to whatever container we want without restarting it and that's fantastic.

A

So I haven't really had a lot of chance to play with this. Yet it's currently in alpha, I'm really looking forward to it stabilizing.

A

So I can use it more because you know it's just good to be so so great to to be able to add these containers so easily and without disturbing the crime scene, um and you can see, there's a command of how you might use it. um Basically, you just give it a pod and a container image and it'll add that container image to that pod. It's it's really that simple.

A

uh So, in summary, uh network debugging is is hard, but a lot of good tools exist. If you, if you know about them, um lingerie tap, is always kind of my first line of defense uh just because anytime, I'm using linkery, it's just very available and it's very convenient and gives you a lot of data very quickly about what's going on at a high level, so for digging deeper, uh I use debug sidecars, and these are really good. If you're, you are planning ahead.

A

If you know in advance that you're going to need them, you can add them. When you create the pod and then is there collecting data the whole time or is there available for you to use? You don't need to restart it when when you trigger the bug, but if you don't plan ahead- and you don't know you need it in advance- it won't be there or, if you're kind of debugging, something that happens in the wild.

A

You won't necessarily have that debug sidecar, and so you have to fall back to doing something like ssh directly into the node uh and using ns enter to to look inside the network name space of the pod and see what's going on, but hopefully in the future, as ephemeral containers stabilize we'll be able to use that instead and that will be a lot easier and we won't have to worry about planning ahead as much and we'll just always have that available. So that's uh something something to look forward to.

A

So that's all I had to talk about today. I hope that you learned some. You know new techniques that that might help you in your own debugging uh or at the very least. Maybe you learned something about the way that pods and network namespaces work when you can appeal back the the illusion.

A

um So thank you for attending. Thank you for listening um and, like I said I work on linker d, we're a growing project. We love contributions from the community and people getting involved and joining us. So uh please find us on slack or github and and come participate. It's a really awesome project thanks and that concludes my presentation.

A

All right, um thanks for watching everyone, I think it's time for some q and a's there's already some great questions in here. So I'll go ahead and get started.

A

So there's one question here from deepak which says: how do you avoid not disturbing environment in production where you must restore the environment while you can continue the deep dive into post-mortem?

A

um Yes, that's a really tricky question and I think that depends uh a lot on kind of the specifics of the scenario you know. How critical is that that you get back up and running, you know quickly, and so I think all I can really say is that you want to just gather as much data as you can. You know wow while the system's in that state, um so that you have that data kind of available to you when you go to reproduce this in, like a staging environment or a test environment.

A

So the more information you can gather you know from uh you know these tools, if you have them available like tcp, dump or or you know, picking up the logs and the metrics. uh That's to help you more accurately reproduce that scenario in a controlled environment.

A

Guy asks: how can you correlate messages using tap? So that's a great question. So when we were looking at that uh linker d tap stream, we were just seeing the communication kind of between two services, um and so often you know that doesn't exist in a vacuum.

A

You know, one request will trigger a service to send it to another service which will send a request to another service and so on, um and that's where a tool like distributed tracing is really helpful, because you can kind of follow a request as it travels through the system and see all the things that branch off of it and everything that it triggers so distributed. Tracing really is the right tool for that job.

A

um Tap itself doesn't really have a way to let you correlate these requests, um but I usually don't use it in that way. I usually look at you, know a single specific service, look at its inputs and look at its outputs using tap and try to figure out. You know whether the requests and responses that it's sending uh are correct, rather than trying to like trace through the whole system, because I'm usually trying to narrow down the investigation. At that point,.

A

uh Camilo asks, together with the message bodies: are you planning features related to use it to replay flows, to run simulations to reproduce bugs yeah? This is this is great. This is really cool, um so the idea here is that you know once uh linkery tap is able to emit not just metadata but full request and response bodies.

A

Theoretically, we could record those bodies and then replay them so in order like basically do like a recording of some traffic and then replay it at a later time to reproduce a bug or to you know, use it as part of some kind of integration, test. Suite and yeah. uh That's definitely been something. That's on my mind. I think that would be really really cool and definitely something that will be possible once once. Bodies are supported, um yeah so for sure.

A

um Let's see uh jan asks how much history is being kept so with linkedin tap, uh there's no history being kept. It's all kind of just live data, and it's just streamed out to you know either to the web dashboard or to the cli, um and so at that point it's up to you whether you just kind of want to you know, consume it live or whether you want to you know pipe it off into some kind of storage and- and you know, store it somewhere to review it later.

A

It's kind of up to you as the consumer of that data. It's not, you know automatically stored anywhere. It's just totally live data.

A

uh Let's see, there's a question here from armin uh who asks: how does the debug sidecar work with non-privileged, pods or containers? um The answer is that it doesn't. uh You definitely do need elevated privilege, privileges to run these kind of uh network debugging tools.

A

So that's. I guess another downside that I didn't really talk about or other restrictions that you do need kind of elevated access to to run these network level tools.

A

uh Let's see uh ma asks if it's possible to get a copy of the slides. I think the slides are linked uh in shed ins in sketch sheds. um So if you go to the uh the page for this talk, you should be able to download the slides there.

A

uh There's a question here from blase asks whether uh ephemeral containers resolve the requirement, and yes uh they do. uh That's kind of the great thing about ephemeral containers is that they resolve the requirement of needing to restart the font, so that's kind of what we've been after this whole time, and so uh I'm really excited to to play some more with ephemeral containers as they stabilize they'll just make using debug sidecar so much easier because you don't need to kind of plan in advance. You don't need to have the foresight that you need to.

A

You know add this container before before the problem comes up.

A

All right, I think, that's, uh I think, that's most of the questions I'm going to be on in the slack. uh So if you have more questions that I didn't answer here or if you think of something or if you just want to come, say hi I'll be hanging out in the slack for for a little while. So please come by, say hello and and I'm happy to chat more about non-forensic or sorry non-destructive forensics.