Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 29 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: There's a Bug in My Service Mesh! What Do You Do When the Mesh is At Fault? - Ana Calin & Risha Mars

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

There's a Bug in My Service Mesh! What Do You Do When the Mesh is At Fault? - Ana Calin, Paybase & Risha Mars, Buoyant

A service mesh is an increasingly necessary tool when running and debugging modern applications. But what do you do when there’s a bug in the mesh itself? Paybase offers the most flexible, developer-native, API-driven solution for payments, compliance and risk. They use the Linkerd service mesh to process all requests that come through their complex system of microservices, where it is highly useful for out of the box gRPC load balancing which allows Paybase to scale their application. In this talk, Ana and Risha will talk about different Linkerd bugs that Paybase encountered after deploying Linkerd to their staging environment, and how they worked with the Linkerd maintainers to track them down and squash them. This talk also explores the relationship between companies that rely on open source software and their interactions with maintainers in the path to getting bugs fixed.

https://sched.co/UaZB

A

How many of you run service meshes how many of you have run into a bug in your service mesh great? This talk is for you. So, as we all know, running a distributed, application can be messy business but you're. In luck, we're gonna tell you about how to find and fix bugs in your service mesh, so this is Ana as you've heard she's a systems engineer at pay base, she's really good at stacking kubernetes blocks.

A

Weight, your bikes off. Did you turn it on in the thing.

B

All right, sorry about that, it shouldn't be a talk without some sort of debugging of some sort, so this is Risha she's, the software engineer at boyand and she's also maintainer of linker D and she's going around the world searching for her twin. So let's have a look at what we have in store for you today. So what's cooking today, we're gonna start by talking about mushrooms and don't worry, everything will make sense. Then we will talk about meshes and specifically linker D.

B

Then we will talk about mysteries, otherwise known as super obscure bugs, and we will look. We would put those mysteries into the microscope to figure out their intricacies. Then we will put our mastery in the showcase and and will hopefully give you some tips and tricks on how to troubleshoot better so I said: I'm gonna talk about mushrooms, so here it is.

B

My boyfriend recently became absolutely obsessed with mushrooms and I, can't make him stop talking about them, but because of that I, and that mushrooms form symbiotic relationship with trees in which mushrooms get sugar from trees and trees, get water and nutrients from mushrooms, and that got me thinking about how a real relationship between users and maintainer of an open-source software should be this. By the way it's called nested, mushroom and I thought that was really cute, but going back to our open source project.

B

When we're interacting with an open source project, we should think of users and montana's as two different groups of the same team. We normally have the tendency to think of two groups against each other, but not that's not how it shouldn't it should be, because an open source project grows based on user feedback, user testing and user contribution when you're communicating with as a user.

B

When you're communicating with someone on an open source project, it gives the opportunity to learn and better understand what you're trying to do, and also it can give you opportunities like talking to Keep Calm. In fact, the only reason why I'm here today is because which are contacted me to do a talk with her, otherwise I didn't plan to be here also, some of the open source project out there have not funded at all, and they don't have any sponsors, which means that the people who are maintaining them are basically doing charity work.

B

So it's nice to be mindful it's nice to be careful of how we communicate communicate so in on blame, free approach and being nice is nice, and we should all remember to talk to each other like we would if we were in front of each other, even when were behind the screen, and with that I'm gonna hand over to fish at the last bowling could be great.

A

So what is linker D linker D is a service mesh and if you come here to learn about what a service mesh is it's an infrastructure layer that is responsible for services service communication, so it sits between your application and kubernetes and it's responsible for the reliable and secure delivery of requests. So imagine you have your application it's running in some pods when you install incurred Eve.

A

What you're doing is adding a series of lightweight proxies that run as a sidecar container alongside your application and traffic is transparently proxied through linker D, without your application being aware that this is happening, and this addition of proxies that run alongside the application is called the data plane and you can see the data plane at the bottom of this diagram at the top is the control plane, which is a set of services that let you interact with the proxies and also get data from them.

A

So because linker D sits next year application and it sees all of the incoming and outgoing traffic, it can do really cool things. You can provide you with a lot of features like zero config. You troll pls, distributed tracing latency, aware, load, balancing and a lot more and Anna is going to tell you a little bit about pay base and what features pay base uses. Thank.

B

You so pay raise is an API driven payment services provider. Our customers are marketplaces, geek, sharing economies, cryptocurrencies or basically any sort of business that has a very sophisticated payment flow and normal financial institutions cannot handle that flow. This means that we are in a highly regulated industry. We have a banking license, which means that we care about availability pretty much all the time, and we also care about monitoring and knowing all the time what's happening in our cluster. So why did we choose linker D? Well, 80%.

B

More than 80% of our platform is built on top of open source software, which means in general that we care about the relationship with this open source project. Also, a hundred percent of our platform is running on top of kubernetes and our application is a distributed monolith that is deployed in a microservices architecture, so the application itself in our clusters.

B

It it's made out of more than 50 micro services and because the the services talk to each other web G RPC kubernetes, does not provide by default G RPC load balancing, which means that if we need to serve more customers- and we need the scalability- wouldn't be able to get that without a custom solution and that's where linker D comes into place. So the main thing that linker B gives us is the G RPC load balancing and therefore scalability.

B

There are other features of linker D that we, like most of them, are around tracing and metrics and things that can give us that visibility. We care about so.

B

I've started implementing linker beam at the beginning of the year in our clusters, and it was a bumpy ride because our application is ridiculously complex and, let's be honest, maintaining a service mesh is not easy either. Alright. So in the first diagram on this picture, you can see the payload of a request that application was expected and on the highlighted square you can see how some of the headers should be and then on the next diagram, you can see what happened to that request when it got passed through linker D proxy.

B

So before so when the application was mashed, and you can see that the headers that work were mutated and because of this, our application didn't know what to do that. So we started getting some really obscure, 502 protocol errors, and it was quite a journey to even figure out what the issue was. So I did my best to come up with a framework that hopefully fully can help you also troubleshoot in the future pattern so troubleshooting before we do that. I want to start with this code by Brian Redman.

B

That said, you should be warned that being an expert is more than understanding how a system is supposed to work. Expertise is gained by investigating wire system doesn't work; in other words, failure is desired if we ever want to develop a software engineers, but also to develop a better system.

B

Also, it's really hard to troubleshoot, if you don't have at least a basic knowledge of our system of your system and how the different requests get routed through a system in the Google Azhar ebook there's this super cute diagram of the different steps of troubleshooting and they look very straightforward, but that's not what I've experienced so they define the problem report as how did you find out that something was wrong, so whether you are on calling your page in the middle of the night, but someone started screaming at you that something is not working or simply you're trying to configure link Rd and you started getting 502 errors, and the next step is triage, which is basically deciding whether this is worth your time.

B

So what kind of severity and based on the issue? How are you gonna pack a lip and then the rest of the steps are basically how you're gonna go around fixing your issue and, as I said, this is very linear, but in reality it goes ups and downs and ups and downs and from one step to the other. So what I thought about is what are those questions that you want to ask yourself when it comes to troubleshooting service Mash?

B

So so I did this for link Rd, but I think he can take this for other service matches as well. Okay, so you have an error.

B

So the first thing you want to ask yourself is: are you sure this is a service match related error and if the answer is no, what you want to do is and mesh the whole thing and if, after a match in the whole thing you still the you still get an error, then it's very safe to assume that the issue might not be related to the service match and it's something in your application, in which case you should completely forget about the service match, go and fix your application and then come back and try again.

B

However, if after you've a mesh the whole thing, the application behaves but expected, and you don't see any errors, then it's a safe bet to assume that you're seeing a service match related issue, in which case the next question is. Could this be a configure so we're all humans? We all make mistakes we'll make whole sort of misconfigurations, and if the answer to this question is yes, then you should double-check. All of the configurations have another look.

B

Have another look at the dog dogs and see what happens and if you're, ridiculously lacking I think it happened to me one and then well-well banging you've solved your issue. You can go on with your brain, however, if you're not liking. The next question you want to ask is: is anyone else having the same issue, and this is where you go and check all of those communication channels that are available to you?

B

So you know slack, github, Google and so on, and if the answer is yes, someone else is having the same issue then ask yourself: is there a fix available and if the answer to that is no, then can you fix it and if you can, please go ahead and be the hero for the day become a contributor and save the day. However, if you cannot, this is the time where you comment on the issue and wait for additional instructions.

B

If nobody else is having the same issue, then you should think about trying a different service mesh version, so sometimes different service mashup versions have different bugs between them. Sometimes a newer version has fixed the bug or a newer version has introduced a new bug so being able to try.

B

Different versions is important and if you haven't had luck with the different version again, can you fix the bug and if you can open up here, if you cannot, this is the time when you file a completely new bug report and we'll talk about the kind of detail you want to introduce.

B

So I've also mentioned that it's important to have some sort of basic understanding of your system and at the beginning, I said that our system is made out of 50-plus microservices. This is a super super simplified diagram of that system without link our team and the way it goes is so you have your end user request hitting your domain and assuming that this is happening in Google Cloud.

B

That request goes to a Google front end and then it goes to a Google load, balancer at layer, 3 and then finally, it hits your kubernetes cluster, your DK cluster and usually that happens through an ingress controller. We picked nginx but linker D supports traffic and others as well, and up until now the traffic has been encrypted and when it hits an GNX it passes through, and it is passed to the next service as pure HTTP and I also mentioned that our services speak to each other via our PC, and this is what that happens.

B

So the reason why this one is important for us is because you're seeing those errors, it wasn't happening straightaway, like you, got the request and then at the service B, you get the error. It was happening somewhere, bounder stack, so understanding the different connections between the services and where it was happening and being able to have visibility was really important. So, as I said, this is very simplified. Now the system with link Rd looks much more complex. So again, your request hit your cluster, but this time because you have link I'd be injected.

B

Instead of it hitting nginx directly, it hits the sidecar container of PNG and export and link Rd proxy will take the traffic pass it to Angie and acts as TCP encrypted, and the response from Angie an X will be passed, darkus, decrypted HTTP, and then that request will get passed to the next sidecar container of service, be secured gr PC, because a linker d proxy will use whatever protocol your application is using, and here the two-way connection between the service pack and fork happens at GRP sea level.

B

And again the same thing happens again when you pass it on to the service I've also spoken about filing donation and I would like to. Please ask you to always follow the recommendations of a bug report, because it's important for the maintainer to fully understand what you're trying to do and what everything you've tried so make sure to attach lock, outputs, make sure to assume that to don't do not assume that people know your system, and you should always be as detailed as possible include everything.

B

You've tried and that's versions of literally everything, and this is not just linker. The related advice. This is open source like universal advice and with that three Rachelle's gonna tell us more about relaxing and fun stuff yeah.

A

So three are just maintained. Errs we get a lot of issues, thank you all for filing them and at the end of the day, we have to figure out what issues are important and what we're going to work on for the next week. So with Lync ready, we have a weekly meeting where we sit down. We look at all the new bug reports and also ongoing books that haven't been fixed yet and we have to decide what to work on that week. So how do we do that?

A

Well, there's two things, one of which is reproducibility. So do we have enough information to understand the bug and what's happening and do we have a test case? Ideally it's a small test case because it's really hard to reproduce everybody's system on our side without access. You know to your whole architecture. The second thing is impact or severity.

A

So how bad is this bug and how many people are experiencing the bug right now, so things you can do to help us would be to comment or thumbs up on the issue, and that will give us one place to see. Oh, this is affecting a lot of people. So when we get bug reports and we're trying to diagnose a bug, it could be in many different places. As we've discovered systems can be complex, so the bug could be in the application it could be in the applications dependencies.

A

It could be in the linker D control plane, which is written in go. It could be in the go dependencies. It could be in the proxy which is written in rust. It could be in the proxies dependencies, it could even be in kubernetes or it could be in the Linux kernel, and this has happened so there's a lot of places. We need to look at, we got Anna's bug report and it had proxy and application logs and we saw protocol errors as she said, but that wasn't enough to tell us where this was happening.

A

So we went into linker t-tapp, which is a linker D diagnostic tool that lets you tap into a particular resource and see the application requests so see whatever the proxy sees at that time. But this still didn't tell us enough information. So we go a level deeper into the TCP dump. Here we started looking at TCP packets, and here we saw some headers were being split across multiple frames, which was nothing we usually see. Some people are looking confused. Let me explain: headers frames, there's some HTTP to background that we need here.

A

Okay, so there's two things we'll need, one of which is how does link Rd use HTTP to the second is a little bit about how HTTP 2 works and you'll hear me sometimes say: h2 instead of HTTP 2, because saying HD, video over and over again is really hard. Ok, so first thing h2 in the linker tea service mesh, so Anna had explained that when service a wants to talk to service speed, the traffic doesn't go directly.

A

Instead, it goes through the proxies alongside service, a and service B, and that proxy to proxy communication is using h2. Okay. The second thing we need to know is h2 background. One of the really cool features of h2 is multiplexing, so having several requests and responses being sent over the same TCP connection kind of at the same time. How does it do this? So each request or response is a message that gets broken down into units called frames and the frames.

A

So each request is sent as a stream of frames and the frames are kind of interleaved with one another and that's how they all get sent each frame has an stream ID that will tell you what requests the response. It's a part of, and a frame type which will tell you what type of data the frame is carrying, and here you can see two frame types, one's a data frame and one's a header is frame.

A

The headers frame, as we might guess, contains headers, and if there are too many headers to fit on that frame, you can send another frame type called a continuation frame. You can send more headers in there. Ok, the next thing I want to talk about is header compression in h2, so h2 uses a compression format called H pack and one thing one way that it compresses headers or sends less data over the wire is that it maintains.

A

So we have the client and the server each maintaining a table of headers and instead of sending the whole header over the wire it'll just send the index of the header in that table and that tables actually two tables a static table of commonly used headers and a dynamic table that gets updated with requests on the connection headers on the connection, so you can see like in the bottom left frame, there's a bunch of headers that get sent, but then by stream.

A

Three we've seen those headers before so it might just send the index in the table and it will only send new headers so back to anna's original blog. These there's a lot of headers, so we might suspect a continuation frames in play and some headers are repeated and in fact, blog number one spoiler alert. There were more than one bug bug. One was that our code was panicking when a continuation frame contained a repeated header bug number two was that we were looking up headers that had been evicted with the wrong index.

A

So again, this bug was pretty deep down in the stack. It was in the h2, rust library that we use, but we did find the bug and we fixed the bug and the user was happy yay, that's great. Every maintainer wants that, but a hero's work is never done, so we always use interactions and bugs like these to maintain or to improve diagnostic tooling in linker need. So some of the things that we have as part of this or ongoing work are a debug side.

A

Car container that you can deploy into failing pods that just has a bunch of tools pre-installed that will help. You grab the right things for us to diagnose the bug we improve linker DTaP, so you get even more information and another thing was a lot of improvements to the rust libraries so better tracing, so that we can see when bugs occur, deep down, we can figure out where they are, and it's going to tell you about more useful commands for troubleshooting.

A

B

On our side, we have to do quite a bit of back and forth, providing more information in order for link Rd to find out what the issue was. So some of the commands to found useful so, first of all, increasing the log level, so by default, clinker de proxy comes the patched with warning level. So you can you have the command here to increase the to debug level, and all of these slides will be available, then being able to skip inbound and outbound port and see how the application behaves.

B

When you do that- and this is important to know when you're doing white listing in your application. So if you're doing that, you want to skip inbound port so that the the original IP gets passed.

B

All the way to your application, then enabling the linker, the debug container, which is an additional sidecar that provides you with TCP done atom, T, shark and other useful commands, then actually running TCP dump dump in debug container, which allows us to analyze the raw packets and see in a different detail, what's happening as rich I mentioned, linker D tab, which allows you to listen to the traffic stream for a specific resource, as it happens, and something we found particularly useful in the very first versions of linker D was chip CTL table quads, and we found this useful when, when mashing an application, we were getting a lot of latency so being able to compare how how long the application and was taking to deal with requests when it was matched versus when it wasn't, was very important for us.

B

So to sum it up, there was more than one bug. There were many bugs and it was interesting to work with linker D to find out how everything fits in together and also solve them, but all of the bugs got fixed fairly quick because we were able to provide very detailed bug, reports and I had space to to test properly. So this is very important. If there's any managers in the room, you should let your mad.

B

Are you Engineers test properly, so that means testing with and without link I'll, be testing different versions of different things, all the way to the infrastructure level, I've seen a lot of bugs on the linker, the repository where people say: hey I'm having this issue, and then you guys come back with yeah, we're aware of this issue. We think it has been fixed in this particular version. Can you try- and the answer normally is sorry. I think I'll have time, maybe sometime next month to try it.

B

We also use the locks suggestions in the issue template and we try to be as detailed as possible and explain how our application is working as well.

A

Thanks, we hope that you will take what you've learned here today and go forth and have a productive and symbiotic relationship between users and maintainer x'. Thank you.