Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Distributed Tracing in Serverless Systems - Nitzan Shapira, Epsagon

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Distributed Tracing in Serverless Systems - Nitzan Shapira, Epsagon

Serverless and FaaS naturally fit microservices architectures. Observability of such systems is very complicated, since each microservice is separated and working asynchronously from the others. Distributed tracing is a key approach to understand such systems. In serverless, there are new challenges and opportunities which make distributed tracing a very interesting and useful technique for high observability. I will go quickly over the history of tracing and the popular tools. I will focus on the key differences between observability using distributed tracing in generic microservices environments compared to serverless. Examples from the popular cloud vendors will be shown, including full visualization of asynchronous transactions in a highly distributed serverless system, and detecting business flows across multiple asynchronous communication resources (e.g. SNS, Kinesis, and more).

To learn more: https://sched.co/GrXp

A

Hi everyone I mean it's an Shapiro from Epsilon and I'm gonna talk about server less than distributed systems and distributed tracing in subsystems.

A

So a little bit about me, I'm software engineer in the last 12 years, I've been doing different things from mostly around cybersecurity stuff, like reverse engineering, networking, embedded systems, a lot of infrastructure related things in the last year and a half I'm one of the founders and the CEO epsilon, and so we are a company that is helping people adopt, serverless, helping them understand, what's going on, their production, troubleshoot monitor, understand, cost and so on and I'm from Tel Aviv in Israel.

A

So the thing we're gonna talk about are, first of all, we start with Cerberus so usually I'm, giving these talks in serverless conferences. So the people are familiar with it, but I'm gonna give a bit more introduction about what I think service is it's more than just function as a service. It's actually gonna be connected to the things you talked about in this conference. All the time like containers and so on. It's more about what service is, as a concept is and then we're going to talk about observability for service.

A

So what do you really want to know? What is important to know? How do you do it? How is it different than what you are doing for containers, for example, and from there we'll go into distributed tracing as one of the main tools and technologies that can help? You really understand these kind of complicated systems, and hopefully my hope is that you can take some of it to your day-to-day job and start experimenting with serverless in your work and see what you can achieve with it and hopefully develop much much faster.

A

So, let's start with service, so service is not really a word, as you know, it's more of a buzzword that everyone tries to give different meaning. For so the way I see service is very different than just function as a service service is the concept that enables you to focus more on your business logic without managing infrastructure, and this way iterate faster and ship software faster and be more efficient and function as a service is one part. So it's one way to bring your code and configure it and let someone run it for you.

A

Usually it's gonna be a cloud provider, and then you can see today's services that are containers a service, so you can bring a container in a similar way without knowing even what host its running on. So this is the way you can bring your code to the server list vendor and then you use manage services.

A

So this can be databases, storage, api's events and triggers that connect it all together to create a service application, which is where you really get the benefit of service, and you can run a completely service application and focus more on your business logic and, as you know, service today is mostly happening in the cloud and AWS specifically is the most popular vendor. So AWS has lambda, but as your Google IBM Oracle, they all have their own services around that, including independent vendors like bananas, for example.

A

They are sponsoring here and have their own function as a service platform that is supposed to be much much faster, but then you can see, of course, open source projects like a native and open fast, that you can also install on premise or you can install in the cloud and they supposed to give you the same experience with several isn't the best developer experience in my opinion, service is actually mostly happening in the cloud because that's where you can actually scale with no limitations and you actually can pay per use, so I'm gonna focus mostly, and this talking the examples are gonna, be mostly from AWS, because this is the most popular service for service people use today, so apologies in advance, but it's actually the same thing that can be applied to any cloud provider or any service framework.

A

So the benefits of serverless. First of all, the financial benefits paper, use it's very simple to understand. It makes a lot of sense because if you look at the right side, this is the utilization of a server in a company. So usually, there are pics, so the servers have to be much more powerful than the average usage. So in many companies the utilization is 10 to 20 percent, so basically 80 to 90, and they pay is for nothing.

A

So if you just pay for what you use, you can save a lot of money and it makes a lot of sense. You can also be in kind of a risk. I'm gonna go into it later, because you don't know how much you're gonna pay.

A

But if you do it right, you can actually save a lot of money, but in my opinion the most important benefits are actually the auto-scaling and what we call low ups, which eventually lead to increase, developer velocity, so developers will be able to ship software faster and iterate faster, and the organization is going to be more efficient and I'm hearing today, companies even very traditional enterprises. That say we want to be fully selves less in X years. Every new application we do now in the cloud is going to be service and so on.

A

So companies really like this idea. So it sounds all good if you could run it, it will be amazing. I mean you would enjoy many many benefits, but there are limitations so those functions they are limited because by design they are limited because they need to do those complicated things in scale. So they have limited resources.

A

Memory running time is limited and basically you have a very strong incentive to choose lower memory, because then you will pay less so then, how do you know that your function will not get a time off and the running time is well, they can run up to in a double slam that it was five minutes now they increased it to fifteen minutes. Still it's limited and you get the fact that it's titleist. So it's it's good it's by design, but now you have to think about how do you design stateless applications?

A

Where do you put the state? It should be in a database, storage and so on, and how do you fix problems when there is no server to connect to? How are you supposed to install something that will give you an idea? What's going on? So that's another challenge and then you get cold stars. You got concurrency limits.

A

These things are improving over time, but you have to be aware of them when you work in service, but eventually, if you do it right, you can really enjoy it and run servers in production in very high scale in very low cost and companies are doing it today, including us, and the main property is for service. Applications is first of all, it is a micro services, design pattern, you basically take small components and you connect them together.

A

The main difference is that its first of all, it's event-driven, so it's not just functions according to each other. It can be events that trigger the function. For example, in the right side you can see API gateway triggers. The lambda puts a file into a storage to use another lambda. So suddenly you have those underlying elements that trigger your code that are not part of your own code.

A

So this is what we mean by event-driven architecture and it's also very distributed, because every every function is limited, so you kind of have to spread it into smaller pieces, and then you get those highly distributed, applications and also you're, very encouraged to use API such as wheelie or stripe or zero. All of them can be used and will give you more benefit because you don't really want to do something. That's already out there and your functions are limited. So you won't really want to focus on your business logic.

A

So this is the properties and just an example, from the reinvent conference two weeks ago, a real application from HSBC, which is a company. You all know that's running in production. So it's not just something that people play with people are actually running these things in production and let's talk about challenges and why do they exist? So this is an example by a young boy who is one of the popular bloggers in service and he actually took a monolithic application in the company.

A

He worked for and migrated it into a fully service application and take a look how it looks before and after so obviously, this is just the design we had on paper and then he had to deploy it, but the main problem. The main challenge in service applications is just the complexity of from a software design perspective. They are just more complicated. You have many more elements, you have events, you have distributed events.

A

So how are you going to understand what's going on there so think about taking this thing, deploying it to the cloud or to any service framework? And how do you know what's going on? How do you know if it's working? Are you really gonna look at functions and their logs, and is it going to be enough for you? How do you know the performance for the end-user, the experience that he's getting, what about cost? What are you paying for? How is it split across the different services?

A

These are questions that are very difficult to understand once this thing is in production and the community actually agrees the last to service this year and the previous one show that the and monitoring are the biggest challenges and it's actually not getting any better, because the application just become more complicated and the basic tools just are not suitable for this kind of applications.

A

So let's go into observability and see how it applies into service and what we can do about it. Sorry, so we're in general, it's not different in service. If you have an application, you want to be able to do the same thing. First of all, you want to know if it's working, so that's a tough question. What what working means is it zero levels? Is it 99%? Success is below 100 milliseconds latency. Is it $1,000 total cost? What what? What is the good health?

A

So that's the one question that you want to answer and you want to be alert that if something goes wrong and then you want to be able to spot problems, hopefully as soon as possible and then fix them quickly, so your customers get the best experience. And finally, you want visibility into what's going on, so you can make decisions for optimization of the performance and also in service. It will observe also optimize the cost because you pay for use.

A

So these things are eventually what you want to have now, let's see how they apply in service, and what can you do to get them? So, if you want to know if your system is working, what what can you ask yourself? So, first of all, most people think of service as functions. Ok, I have 50 functions, they deployed them. It's my code. Let's see how my functions are doing so is this enough, so basically it makes a lot of sense. Just look at the function. Metrics see the errors.

A

The failures make sure it's working, so first of all it functions are important and identifying errors. Time outs out of memories called stouts are very useful to easily find problems in your applications. So just by measuring those metrics, you will be able to spot problems easily. But if you think about this kind of application, is it really just the functions? Is it enough for you to look at the function, metrics and really understand the entire story, because you can see that a request comes in triggers multiple functions.

A

Api's events, external API, is: maybe you get time outs? How do you? How do you really know that it's working? It's not just functions and it's actually functions, but in addition, api's events triggers and what we call transactions. So a transaction would be an event that starts triggers multiple things on the way and ends which is directly related to the customer interaction with the system.

A

So this would be a better way to understand the system and to tell the entire story and if you think about troubleshooting so think about, if you had an elbow in one of the functions now, you need to find out what happened. But there were five functions that came before and they all communicate with each other pass data to each other, so just by looking at logs and logs and logs, this is not the right tool to troubleshoot a distributed system and I.

A

Think people today already know that distributed systems cannot really be understood well, just by using logs. This is why distributed tracing came up so eventually in service. The thing is that the events are asynchronous, so you have many a synchronous, invocations and events, and you need to somehow connect them together to create a distributed trace and what we call a transaction just an abstraction. We look, something like that came from request. Go goes all the way through multiple events and let's take an example.

A

So if we have this application, what eventually we want is to say: okay, we had a function, but it actually emitted an event and triggered another function and another function and is all connected. So we can trace back and see exactly what happened, and the second C thing is to be able to go outside my cloud providers domain and go to external API such as or zero even another cloud provider, or even from on-premise to the cloud or vice versa.

A

Tracing this API is very important in servers more than it is in containers, because, first of all, every API call you make can call cause your function to timeout. If you depends on your running time limit. Second, you pay every time you make a call you pay for the time. So it's even more important than before to measure those API calls.

A

So now we are going into distributed tracing. So essentially it's been. There will be a few talks in this conference about distributed tracing and I. Think the whole of the ability world today for micro services is going towards distributed tracing, because this is how systems are built.

A

New systems are usually built as microservices, and the point in distributed tracing is to be able to tell the story of a trace that comes into the system, goes through multiple services, either via direct communication or by emitting events to an underlying infrastructure, and eventually this trace goes on and on and on until it finishes, and this is really what happened in your system from a business perspective, we try to troubleshoot something that happened.

A

Distributed tracing will really give you a much better sense of what happened and when you aggregate this data over time, you will get a much better information about how your system is performing and not just simple metrics of functions or CPU of the host and so on. So this is the point in tributed tracing and, for example, Yaeger is a very popular tool for visualization of the tracing data. So if you have the tracing data, the tracing expense, you can put it into yogur, and then you will see those waterfall charts.

A

You will see how much time everything takes, and it's very easy to understand where the bottleneck is, for example. But how do you get this data? So if you have it, you can put it in this kind of tool. For example, how do you get? How do you connect the dots together? That's actually the tough problem so distributed tracing can go different ways, one of the popular frameworks that, of course it's a part of the CNC F. It's open tracing.

A

There is also open sensors and these frameworks enable you to create traces to build, trace pans and by implementing them in your code. You can get those results which are pretty nice and then use something like yogurt to see them, and this is a very good approach and companies put a lot of effort and implementing these things in their system in their micro-services and eventually they get great results. And now the problem with this approach is that it's not really a problem, just the reality that it's manual, you have to put a lot of effort.

A

You have to learn, you have to implement, you have to maintain, and eventually you can spend a significant amount of your developers and engineers and architects and DevOps time. Maintaining this thing and writing new things this way. Sometimes you don't have a choice. So that's what you do and but again, let's talk about some other options because, instead of the less you have a different world, it's not completely generic. You live inside the world of the silver. Let's fly mock you walking, so you can live inside the ws.

A

You can live inside K native, but it's not completely generic. So you can actually do some things differently, so think about manual. Does it really make sense for servers if you had to go into every one of those functions and implement something manually? You have you have thousands of them. If you had 50 containers and micro-services, maybe now you have 5000 nodes in the graph. So does it really make sense? I mean the whole point of going service is to work fast and to iterate faster, develop of velocity.

A

So if you say okay, it's really cool, but stop! You have to you know. Do this instrumentation and manually and kind of you know it kind of misses the point in a way. You're gonna spend a lot of your time. Doing these things for just to get observability, the question is: if we can do it differently, it will be a great success, because then we will have both the visibility and the observer ability and you can work in server as fast as you wish.

A

So this word is your automation comes in so I'm a big believer in automation in all aspects, from testing to deploying to observability and tracing and generally the world. The entire point of micro services and in the cloud and kubernetes is to iterate fast. So if you can do things automatically, you can benefit much more and eventually just go faster. So what we do in the company is we do distributed tracing for service automatically, so I'm not gonna go inside.

A

How, but the point is that once you have it automatically, you can just go and deploy and then understand. What's going on and I'm gonna give you an example of distributed tracing in service. So let's say you had this simple transaction: it's it's! Actually, it's simple, but it's actually not that simple to understand.

A

If something goes wrong because you have a function that is triggered by an api gateway, so HTTP request comes in, you have another function triggered by an SNS, so this is the first function, publishes the message to the SNS and trigger the second function, and there is also authentication there before and then the second function failed. So now you want to understand what happens. So if you just go to the logs, you can see what happened in the function. But how do you go back one step before to the other and connect the dots?

A

Maybe they are running billion times a day? And how do you match the rocks? It's sometimes just not possible so with distributed tracing. What we would want to have is something like okay show me the problem in this function. So this is a zero division error like any good example, and then you can take a look at the function and see what happened there and then the next step will be to go to the SNS and see okay. What was the message?

A

So you can go back and see the input and one step father will show you the log of that function automatically. So you can say: okay, the authentication failed. So this is actually the problem and looking at the API code, we show you that, indeed it failed and then an API gateway will tell you what was the HTTP request they had on the body. So this kind of tracing can solve the problem in minutes instead of hours or even days. So this is very different than just looking at functions individually.

A

This approach is much more generic and suitable for several applications, and we talked about some of the challenges in service and how to monitor these systems and I'm going to talk about something that is, it may sound unrelated, but I'm going to show you how it is related, which is cost so in service time is money type use means 400. Every hundred milliseconds. Your code is running.

A

You are paying a very small amount, but it can be very big amount as well, and it's a simple formula just multiply it, which makes you really want to optimize your performance, because then you will also save a lot of money, and if you want to identify bottleneck, maybe you will also identify what's causing your application to be 10x more expensive than you expect it to be. So if you want to go in optimize the performance and the cost, you should ask yourself: where do you spend most of your time?

A

Is it on my own code or is it on the API calls that I'm making? That's why it's so important to trace those api's and I'm gonna?

A

Give you an example that happened to us in the company, so we are also server as we have about 400 functions in production in pretty high scale compared to a company our size, and we are using our own solution to monitor our system and one day this was about five months ago, I took a look at the system and saw that we have one function: that the estimated cost for the month was about $15,000 one function.

A

We are a startup company, it's not the best way to spend all the money in the beginning, so I messaged, my partner, my city, oh I, was sure it's a bag or something obviously so I said, is this for real and then it took a look and said yeah. It was real. We had a function that something really bad happened with it.

A

So I was like this and then and then we looked into it and it actually enabled us to find one of the more complicated architectural problem and that caused the big performance problem in our system. So this is how cost can really help. You understand what happened and the way it happened. Just on the core I'm gonna go quick. We had something that connects to the API of cloud watch and gets logs all the time.

A

It was good in the beginning, but it was very, uh not scalable, so every function that our customer has, we would spend another function that will call the logs and after some time we just called the API of cloud watch too many times and it didn't didn't work anymore. It just respond too slowly and those functions started timeout. So we had 5,000 functions, timing out all the time, five minutes forever. So this is expensive.

A

Apparently, if you do the math so the way we found it is actually by looking at the cost, metrics and eventually measuring those API calls will will actually help us identify it much quicker. So that's another way so to understand how service system works. It's very helpful to look at the cost and another example about external API, just a simple example with all-zero in Twilio. If I just make a quick look on the earth zero here, I can see that it's much much slower than the rest of the API calls not because it was zero.

A

We have using goes there, it's a great service, but because it wasn't configured properly so now you have this API call all the time and you're paying for it, and also it may cause your function to timeouts. Again, you have to make sure that every API call you use is is in place, so you can actually scale without worry about anything and okay. Now, if you think about visibility and really understanding your system so function, matrix is a good way cost it another way, but eventually the application has certain business flows.

A

Maybe a user subscribes to your service, a user transfers, the payment via the website and so on, and each one of these user interfacing operation will create a series of events which you can call business flows, and if you know that they look like this or like that, you can actually know how your application behaves on a much more business oriented way. So you it's not just you have ten functions or a thousand functions, but you have ten different flows. One of them happens 90% of the time, and it's not so important.

A

The other one is very important. I want to make sure it's fast. I want to make sure that's where I spend most my money, because I don't wanna spend most of my money on something that doesn't really help my company and again understanding these business flows can really help make decision about optimization of performance and costs, and then you can say: okay I want to focus on this business flow and go inside and see exactly what element is the slow spot and what can I do about it? Is it changing my code?

A

Is it increasing the function memory? Is it changing to a different API and so on?

A

So, just to conclude, first of all, I think serverless is awesome, I mean we are using on it, not because we have to because we want to, and once you start you kind of get addicted to it, because you can really deploy much much faster and companies are using service today to build software faster. Everyone is doing it, including your competitors. Probably so I would recommend at least trying it, and if you want to understand the system, you have to remember sometimes taking the old approach.

A

What works for a server and applying it to into a function just doesn't work. So this is where distributed tracing and other new technologies come into hand, and the last thing is automation, so always think about automation when you go into these new technologies, because you want to use them the right way and you want to save time. So that's all, thank you feel free to email. Me message me talk to me.