Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What's the Performance Overhead? Answering the Biggest Question in Tracing - Gabriela Soria

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

What's the Performance Overhead? Answering the Biggest Question in Tracing - Gabriela Soria, Outreachy

With the increased adoption of microservices architecture, performing tracing is becoming crucial to understanding the behavior of our applications, detecting abnormalities and interpreting transactions. In this session, we'll present the results of performance tests in order to understand how the instrumentation affects the performance of applications. The scope of this research is the OpenTracing Java API, and some representative libraries that use it under the hood, e.g. Java Servlet Filter, Spring Boot and Cloud, JAX-RS and JDBC. We'll be showing tests comparing a non-instrumented application with the same example application using different tracers. We'll include the main concepts of tracing for those who are new to microservices and/or distributed tracing, but you'll benefit even if you're already familiar with OpenTracing. Familiarity with Java is useful but not strictly required.

https://sched.co/MPbd

A

Hello, everyone, thank you so much for coming. This is my first talk ever and I'm so excited to share with you my talk about. What's the performance overhead in racing I'm, Gabriella, Soria I'm, a former outreach intern for CNCs and I want to spend a few minutes to talk a little bit about our Ritchie Ritchie is initiative of software freedom Conservancy and provides three months paid internships for underrepresented people to work in free and open-source software.

A

This talk will cover the results of my internship working with open tracing and younger communities.

A

If it wasn't for the opportunity that our Richie gave me the encouragement of my mentor to person, this talk, the CNCs, diverse scholarship, I'll not be here and I want to say. Thank you. Thank you. Thank you.

A

This is that they're not for dead topics that I'll be covering today fits for the people here that are not familiar with some concepts. I'll give you a brief review about microservices distributed tracing open tracing and Jagger, then I'll, representing how the tests were built, showing an example and presenting the results, and then I'll share my opinion about what should be the next steps for this work.

A

So a brief introduction about micro services, this Martine foulness Fuller's image is probably one of the most famous images about micro services and helps to show the big differences between a monolithic application and one micro service application.

A

Micro services are an architectural style that proposed to have independence, Lucie couple services respondent to a business responsibility. It means that the responsibilities of our entire application are spread into multiple services.

A

This architecture has so many benefits we can deploy more frequently. We have freedom to choose the right technology for the problem problem we are solving, but also this architecture adds some challenges as, for example, understanding the transaction flow from one request through all our system and that time of problem can be solved, use distributed, tracing basically distributed tracing is a method to profile and monitor or our applications. It helps to understand Durst and the behavior of our application and the flow or our transaction, but the most important to the tech performance issues.

A

On the other hand, open tracing is a project sponsored by CN CF that provides an standardized API to allow to write the traces in a vendor neutral way.

A

This means that if we decide to instrument our application today and today, which is one tracer implementation and in the future, we need to change it. We don't have to change anything in our application, except the tracer implementation and Jager. There is an open source distributed tracing system sponsored by CNCs, which implements open tracing.

A

This image shows traces and view address can be thought as direct acyclic graph of response. Each row here is a span. The span is an unit of work and it has the operation name, the start time and the duration.

A

So let's move to talk about the test we built. These are some consideration that we took before building the test. We use a toolkit created by open JDK to build been benchmark test. It is called Java micro, benchmark, harness or th the results that we got after the execution of the test.

A

We process with the amazing tool called gmh visualizer and the scope of the test that we built is just for open release in Java and we hope to expand to another language in the future and we have various benchmark tests and for all that was possible. We try to avoid Network calls in order to have the del. Does the most isolated possible.

A

Here you can see the command for start a new GM age application.

A

This is Ana standard mail in common, and the important thing here is the archetype group ID, which is reference it to the image and with this common we create new project with the required libraries and with the required client's, 4G image, and this is how one of our test looks like in this case.

A

This is a test for open tracing for a spring cloud, and here we can see that is a normal double class with methods, but the metals methods that we want to benchmark have to be annotated with benchmark and also we can put the mode of the benchmark. For example, if we want to measure in the average or the simple time or the throughput.

A

In our case, we decided to separate the modes, because the results that we go separate- the modes words more easy to represent graphic graphically with gmh visualizer and.

A

We have one method with known instrumentation and one method for each tracer implementation that we want to to test. We want to benchmark.

A

This is another important part of the benchmark test, and sometimes you want to initialize some variables that your benchmark commits, but you don't want to be part of the code that your bench core measures and, for example, in this case we initialize the spring context and we assign a viral with the controller that we need to use after in the execution of the test. Also, after the test finish, we close the spring context.

A

And here we have classes extending the state parallel classes that we seen before and with these classes we use a spring profiles to set the property for injector right tracer implementation for each test that we're trying to build.

A

So after we have test, we need to generate the product with this maven standard common, and we run the benchmark with this simple Java common and the results of this are JSON files in a subfolder of gmh.

A

Sorry in a subfolder of it, each project and start with the prefix damaged something and these files are the input for generate the graphics using geometry. So later,.

A

So this is the list of the tests that we built.

A

We started with a simple Java scenario without the with the instrumentation written manually, and this is scenario only test: an extreme compatible catenation like concatenate hello, world, one iteration number, and with this scenario we don't have either HTTP calls or frameworks involved.

A

This is scenario doesn't represent a real use case, but it provides a very good baseline of the teracle limits of the to involve.

A

Then we have an extreme good project. We did the instrumentation manually to and in this project will simulate a bidding process, like generator line-items, notify the customer calculate the taxes, etc.

A

Another test is the spring boat, and here we use the dependency open tracing the spring cloud started and we use the pet clinic example: application officially from spring.

A

And here there is no HTTP calls involved that it means we call in the test to the spring service directly.

A

Another test is the JDBC we use in memory database h2, and we instrument the the application using the library open, tracing JDBC.

A

This application is an example, of course, mana management, and in this example, we call in the test to the direct directly to the spring services and for the last two tests to set of test servlet, filter and jax-rs.

A

We did Network goals that the part of the test executes an HTTP call to perform this. In the case of servlet filter. We use the dependency open dressing, web server, load, filter and in the case of jax-rs, who use open dressing, jax-rs discovery and the tracer in the last two tests is registered in the global tracer, using an application listener and on the startup of the web. Application.

A

So, let's see the results for this simple java scenario.

A

With the the test execute and simple in string concatenation and the test compared our non instrumented, a scenario with instrumentation with different tracers, we can see in the x-axis we can see. We use haystack, tracer layer, tracer mock tracer with without termination on knob tracer.

A

Let's analyze the numbers, the numbers for non instrumentation and note tracer are very close and equals. In some cases.

A

Mock tracer is not a very good comparison, because it's not intended to be used in a production environment and keeps the sponge in in memory that decorates the performance. So for this and the other scenarios, we will compare the non instrumentation with Jaeger tracer.

A

We also have tested haystack tracer, but just as a reference point, you can see in a blue lines here: I'm not showing the numbers in the slide, but according to the numbers, the simple time increases around forty four hundred percent. In this simple scenario- and it's not intended to sorry- it's not easy to see here, because the graph is distorted by mock tracer and haystack tracer, but that's the number, and if we see the other metric the throughput, we can see that the throughput decreases by around ninety percent in this simple Java snart.

A

This case does not represent a realist case, but it provides a very good baseline of the teracle limit of the truth, symbols and the big big overhead here that we see when using our Realtors tracer is indeed a specter and is a sign that the benchmark is same.

A

So, let's move to more real-world scenarios here we can see the results for the spring good application containing a building process. The application is initialized in each test, iteration and using profiles. The right tracer is injected the different test, measured the process of issuing an invoice in a non instrumented, a scenario and instrumented with different tracers.

A

This simple time increases around 200 percent. In this Springwood scenario,.

A

And if we see the throughput, the throughput decreases around 67%.

A

Let's move to see the results for spring cloud, this test used up a clinic, simple spring based application, and the application is initialized for each test and using profiles. The right, tracer implementation is injected and the different tests measure the process of finding a pet owner by ID in a non instrumented. The scenario an instrumented with different tracers, the simple time increases in around 4% in this scenario and the throughput decreases around a person in the spring good scenario.

A

So, let's move to JDBC this test use a simple boot application containing a basic course management process. The application is initialized in each test, iteration on user profiles, the tracer is injected and the test measured the process of getting all the list of all the courses in a non instrumented, a scenario and instrument that with different tracers and the simple time here, increases in 13% and the throughput decreases around 8%.

A

So if we would move to the test that involves HTTP calls like servlet filter for this particular test, we use a servlet, simple application to process again and return an HTML, it sort HTML page with information of the request and a random number and using understo api.

A

The servlet is deployed and the write tracer is injected and with listeners we registered the tracer in the royal treasury. The test consists of making a simpler quest to get a hello world page in a non instrumented scenario. An instrumented with different tracers.

A

The matrix here shows no evidence of overhead pass. The deltas of throughput and single time are no representative, and here you can see the graphic for the throughput for the case of servlet filter and the last scenario is jax-rs.

A

The tests here use on jax-rs, with Springwood application containing a basic course management process. This application exposed jax-rs, endpoint, called cursors, and application is alicia initialized for each test, iteration and using profile. The write tracer is injected.

A

The tracer is registered in a global tracer at startup using a listener, and the test consists of trying dress code to get a list of all the courses in an non instrument that a scenario and instrumented with different stressors and.

A

Here also, the matrix shows no evidence of overhead as a deltas of simple time and throw good are no representative.

A

So, in conclusion, never head comes from the actual tracer and is closely related to how they work on how they are configured in simple java scenarios. The throughput decreases around ninety ninety percent and the simple time increases around forty four hundred percent.

A

In this case of the simple dial scenarios is not a real use case and they were head that we see here is indeed a specter, and this sign that our des is same in a scenario. Study include calls through the frame world like spring would spring cloud or JDBC. On average, the throughput decreases around twelve percent and the symbol. Time increases around four Reapers fourteen percent.

A

I have to mention that these numbers are comparing a non instrumentation, a scenario and with instrumented, with dagger tracer and in the scenarios with client calls through HTTP like servlet, filter or jax-rs. The matrix shows no evidence of our head as a deltas of throughput on single time are no representative.

A

So the next steps, in my opinion, is death with specific, concrete tracers such joggers, or order to understand how much their head can be controlled with modifying the configuration as, for example, sampling or different protocols or different Perkins configs, and this test can be used with the margin of open dressing and open senses to understand if the changes introduced it in the process. Effective performance.

A

Thank you here is the link of the report and with the results. If you want to keep in touch, this is my Twitter. Thank you.

B

Are there any questions.

C

I was wondering you said there were no evidence I've ever had. Why are you doing the HTTP calls in the last two test cases seems a bit odd seems a bit odd to me that there's no other overhead into executing the additional code. So did you consider that it could be a problem of you did not put enough note on the viola network cards on the on the system. You try to measure or have you another explanation. Why there's no overt.

A

I'm not sure about the answer of that question, but I think if we considered it the cost of the test proportionally to the cost of the network and IO.

A

The Delta is not representative yeah, but that can be an improvement to the test. Yeah.

D

So I know for tracing. Usually the advice is to sample like something like one in a thousand requests for actual tracing based on this. Since real tracing typically has HTTP requests, it seems like the only reason for sampling would be to reduce the burden on the like actual tracer. Not it there's really no risk like I guess. Would you say that it would be okay to trace every request since there's no apparent overhead with, and that wouldn't like hurt your applications performance so.

A

All right, I, don't understand the question. Well,.

D

I can rephrase I was kind of wondering so I've wondered if it's okay to sample every request for tracing or if.

A

D

Risky, because that could cause too much overhead, would you say that it's okay from a performance standpoint to trace it every request, yeah.

A

The approach that we use is simple sampling, the older request, but maybe was not the best. Actually that was one of my points in the next steps.

A

It will be important to understand if, with assembly, we have different deltas with different numbers, but with this configuration something all the requests, the overhead of the test with non-human tation scenarios, uninstrumented with yogurt, were not representative. I.

B

Also think that the load on tracer buckets might be problematic.

E

Thank you for talk. I was wondering about the graphs that you showed. What did you show different run? So what was the the x-axis, and why was our graph drawn between the data points? What was the eastern meaning of that? We.

A

Didn't have proxies.

A

We tried to measure the most isolated possible. They were had having one application and instrumented that application and cold calling to the spring service directly one was possible and in a scenarios one that was impossible. We use HTTP calls for the get request or something like that, but this is one of the scenarios that probably we have to stand in the future. Okay,.

E

Yeah I I understand your test at that, but I was wondering: did you consider showing the data in a table with error bars or what is the significance of having the data points for one set up distributors of the x axis if they signify different run instances I.

A

Don't know sorry.

F

A

We we measured the the test multiple times in the in the graph that we we solve one.

A

Let me show one of them. Each of of this points where one execution and we compare with different configuration with multiple executions and the patterns were the same I.

B

Think the question was why you chose this representation instead of just like average and I care of ours.

F

Sodium is there jasira standard deviation as well. So we have a lot of variation between the runs. Then we know we have to look into that. But if they are quite continues are quite linear, then we know that it's consistent. The results are consistent because they're wrong with performance chess. Is you don't know if what you get is really what you're testing? But if you, if you so, if you have high balance, then you might not be testing the right thing yeah. So that's maybe the reason why we have multiple dots there.

B

Hey other questions.

B

If not, thank you again. Thank you.