Cloud Native Computing Foundation OTel Community Day NA 2022, 29 Jun 2022

Previous Meeting

⏯

youtube image

►

From YouTube: The Unexplored World of OpenTelemetry and Service Mesh - Michael Haberman, Aspecto

Description

SAVE THE DATES!
ArgoCon – Register Now | September 19-21, 2022 | Mountain View, California - https://events.linuxfoundation.org/argocon/

KubeCon + CloudNativeCon North America | October 24-28 | Detroit, MI - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

KubeCon + CloudNativeCon Europe 2023 | April 17-21 | Amsterdam, The Netherlands - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. Go here to learn more: https://opentelemetry.io/

The Unexplored World of OpenTelemetry and Service Mesh - Michael Haberman, Aspecto
303+304 (JW Marriott Austin | 110 E 2nd St, Austin, TX 78701)

Speakers: Michael Haberman
Link to join the virtual event: https://www.accelevents.com/e/otel-community-day-na-2022
The virtual event concludes at 1:50PM CDT; In-person event concludes at 5:00PM CDT

OpenTelemetry shines when there are a lot of services communicating with each other. Getting started with OpenTelemetry can be a challenge as it requires a lot of knowledge. Having a high entry point effort, it seems like a good idea to set up a service mesh solution to produce our desired distributed traces. In this session, you will learn about the good, the bad, and the ugly of using service mesh tools (such as Istio) to generate traces as a replacement for OpenTelemetry. We will look into sampling, 3rd party services (e.g., DBs and message brokers), ease of implementation, visualization, and much more. By the end, you will be able to decide whether to use OpenTelemetry, these types of tools, or both.

A

So hello, everybody, my name is michael aberman, I'm from aspecto and today we're going to talk about um two interesting things: we're going to talk well, of course, about open telemetry, but also we're going to talk about service mesh, and I think that um I think that, when we're trying to see um when we combine those two things we're going to see that they are interesting because of two reasons so first thing both of them can produce telemetry data right both of them claim that can help us to monitor to observe our our application.

A

But there is an an interesting overlap that we have between the audience. So a lot of people that are interested in running open telemetry would also use service mesh and then, when you have two things that can work good together with one another and they are the same audience- that's that's a good match. So this is why I was interested in doing this. This talk so um a bit about what we're going to do here today.

A

So I'm going to really briefly talk about what is service mesh, just to make sure that everybody is aligned, but mostly I'm going to talk about where service mesh meets open, telemetry and where open telemetry meets service mesh a bit of other configurations and, lastly, to talk about.

A

um What's the pros and the cons, because you know in any technology decision that you take there the good stuff- and there is the bad stuff and we need to talk about both so um really short service mesh is a networking component, part of your infrastructure that eventually responds about networking and mostly I'm going to refer to service to service communication.

A

So when your service needs to communicate with another service, you're going to communicate through this mess, and I'm going to look into three specific things that that the service mesh can help you with so the first one is going to be with observability, then routing and lastly, on authentication. We try to see how we can benefit when you're, using though things so um here is a very, very complex architecture.

A

We have an older service, communicating with the user service, then they are service and when they communicate with one another, they are going to communicate through a mesh. If you were to go to most websites of service mesh like east geo, like click d, you would see that they would say you can get tracing out of the box. Now imagine going to your service match going to the configuration file.

A

Turning on this magic tracing configuration and everything just works, that sounds really really good and I just want to put it out there, how it really works and what you are getting and what you are not getting.

A

So um we can tell the service mesh to generate spans, and here I have it not here here. I have like a very, very simple application, which I ran before we started this talk, and you can see here that we, I can decide whether to initialize or not initialize the tracer file. So currently what I have here, I have two servers. Communicating with one another through through envoy um and open telemetry is not activated so looking at our example here, so I sent a request.

A

Now, let's go to jaeger and let's refresh it and let's look at our proxy- and you can see here that we can see a trace where I have firstly, the proxy, then I can see that I'm sending a request to the um order service, but I don't see the request to the user service, so I do have spends. I do have a trace.

A

I just don't have the context. I have a broken drop context and this is the truth that I think needs to be told that if you are just enabling your service mesh to start emitting spans you're not going to get tracing, you are going to get traces but you're not going to get the whole tracing experience that you expect.

A

So what you do need to do is you're going to propagate the context yourself right. You need to implement open telemetry, you need to instrument the services and you need to take care of the context propagation.

A

So what I'm trying to say here is that now, if and let me show it- um you know, live so you'll be able to see that it actually works.

A

But if I'll change this parameter to true meaning that now we will have open telemetry running true and it would take a second, then our trace will look like a whole trace right now, I'll be able to see the everything that is happening there.

A

So it's just kind of the this thing that uh um some people may think that you can get tracing from a service mesh, but it's not actually the case because well, you really really need uh to implement open telemetry, and this is a quote from the additional docs saying that hey yeah we do have tracing. However, just a tiny little bit, you still have a way to go.

A

So what I'm saying here is um when you are using service mesh, you are going to generate. uh You are going to implement open telemetry, but the question that I want to ask is: is the other way around? If I am using open telemetry, I already implemented open telemetry. Will it be beneficial for me to get spends from the service mesh right, because I have to have open telemetry, but what I'm gonna? What am I going to get when I'm going to get spent from from istio or lincoln d?

A

So, let's take for at least for my opinion, probably one of the most common use cases that you see with the service mesh and that's routing. We are going to route different requests um when, when we are getting a request, we are going to routely differently, whether it's because we are running an ap test or feature flagging or I'm just running some deployment, a country deployment, and I just have two versions of the same service and I need to route those things.

A

So, if you think of that from the perspective of tracing when you look at the trace- and you see this trace- and you ask yourself well um to which version it belongs, is it belong to the version that I already deployed? Is it the one that is going to be deployed? um This has it. It can be very critical. Let's say that you are deploying in performance improvement and you expect to see a span that now is taking um less time and it's not there. It's taking the same time.

A

Well, is it the new version or the old version, so these things that you may be able to solve um using plain open, telemetry like the using the distro, um but looking at the data that the service mesh produces, could be very very beneficial.

A

So um this is a trace where we are routing the request and you can see those two traces. The first one is 12 span. The second one is 13 spans, and here we have no telemetry data that is being omitted from the service match. um This is plain open, telemetry and when I am going to add the service mesh data- and you can see it right here now- it start to get a bit more details right now.

A

If you look at the first request, you can see here that um here we communicated with service order number two and here was with number one, and this is a great clue for the for whoever is going to observe this trace you're going to understand. Okay, now I understand what's happening now. I understand that there is some kind of decision that there are two versions and I know which version I'm looking at and um let me try and and show it to you like when it's actually running so you see here.

A

This is service version number one and now sending a few requests. Two minute request service too. Thank you service, too, and now, let's look at those and now we can see it in live, so we can see that there is a difference between them and I can understand the difference quite well and if I'll click this one and some thing is happening with the timeline.

A

So I don't know why the timeline is so off, but um it's making it very very easy for me to understand that this is a decision that the proxy took and the proxy decided to use now order number two.

A

So um what I'm trying to say here is without the service mesh data. It would be harder for me to understand this whole trace now. Another interesting use case that we can see in regarding authentication. So when you're talking about routing um this is basically saying once com one service is communicating with another and in the middle we had a decision that we took, but service meshes could be even more inclusive in our application.

A

When the service mesh is responsible of the authentication, they are actually becoming part of the application. So when the client is calling an api call to um to the older service, we want to authenticate the user before it reaches the other service, meaning that the service mesh is. Although it's an infrastructure and a networking piece, it's now responsible authenticating, which is absolutely part of the application layer.

A

So um if I would look at a trace where I'm sending an api, a call, the service message gets that um take the call make sure that the user is valid and only then going through the service. The trace would look like so so I have this a bit of a broken trace right. I don't have a single root parent to this trace, I'm not sure exactly what's happening here and I don't know why the user service was called.

A

I just can see it was called, and I can see that after that we got an api call to the other service.

A

Now, if I lacks my service mesh to emit to limit your data as well, it's going to look much better, so looking at it, let me increase that, so we can all see so now I can see the whole story now. I can see that we sent an api call to get request. We did this async egress thing where we authenticated the user and then we communicated with the rest of the application.

A

So what I'm trying to say here is even though service mesh is the networking pith. It's part of the application. It affects how our application behaves and if we're trying to observe the application- and we are trying to say we can look at traces- we can look at the observability data and we can understand what's happening in the system. Well, it's part of the system.

A

It needs to meet telemetry data, so we can understand the whole the whole thing um so just to sum up the kind of the options that you have with with installing um the you know, telemetry as a whole, you can just go with implementing open, telemetry distal. um It probably would give you I don't know 90 of what you're looking for, and that's that's great if you're only go with service mesh, you are left with a broken context.

A

That's I guess not going to take you very very far and you are going to have both of them. You are going to be super happy now. If I were uh ending this talk right now, we will be super happy, but we need to talk about the real stuff, because everything in in tech come come with a bot right, because when uh we are going to introduce um service mesh and start to to implement traces, bad stuff are going to happen.

A

So, let's, let's talk about those a bit, so I'm going to talk about three things that um are going to be affected, cost sampling or specifically, head sampling and a bit about the configuration.

A

So the first thing that that we can understand is that we are going to increase the overall cost. When I'm saying costs are not, I don't mean like dollars, you need to pay the vendor. I mean the overall cost. Even if you manage it yourself, you are going to generate more spends and just by introducing at least one more or even two more spans for every network activity. You're going to have this could sum up to significant cost.

A

Now, um when you're in increasing the cost, you need to ask yourself: okay, what value did I get? So if the value that you're getting is significant, yeah go for it, it makes sense. So I saw companies um implementing um easter traces and the only thing that they could learn from that is: what's the extra um in latency that history introduced and and that's fine, that's not tons of value, I'm not sure it's worth the cost well depend on them.

A

But when you start to really do things that, like like routing like retries like back off like authentication, then you really need the service mesh to tell you what happened in order to understand the traces and then I think the cost does worth it, but it's every deployment. Every company needs to take this decision for themselves.

A

So this is um the first thing that uh I think it's the bad news that we need to think about. The second one is regarding head sampling. So if you are using head sampling, you are kind of used to that. The open telemetry receiving the first decision once they are getting the request talking about the service mesh. The service mesh service mesh now is before your application and the service mesh is going to take the decision for you.

A

So if you were writing a code- and you said okay in my service, I want to not sample health checks. Races, it's not! It's not interesting.

A

Now the service mesh well depending on the service version, but most world service measures that I know they are only going to give you the possibility to define what percentage of uh traffic you want to sample. So if you are relying on head sampling and then you introduce some service mesh that may go to waste and you can start to to to solve it, you can start say: okay, I won't do like a parent-based sampling.

A

I won't look at whatever the service mesh decided, but that's going to to take you to a whole kind of bad experience and and a lot of work that you need to to do in order to make it actually work. So this is something that out of the box, I think, doesn't really work.

A

um So this is another another area that you need to consider, and the last thing that I would mention regarding uh debate is bad news, at least is that when we are working with with open telemetry, we want everybody to work as we expect with open telemetry with you know the latest specification and follow the semantic convention and use otlp.

A

But the truth to be told is not that's not always the case. So I took envoy, as example, just because many service meshes uses android uh behind the scenes. um So envoy still doesn't support otlp. They do have a pull request for that, um but it's still not merged. um So currently, if you're using it, you need to export the data through a zip key through jager, which is fine, but they are not using w3c context propagation.

A

So now you implemented open telemetry in adobe, 100 services using w3c, and now you said okay, I want to turn on tracing in istio and now you're in a problem, because you need to change it to b3. So it's not something that you can't solve. It's just a lot of management that you will need to do. It's a lot of changes that you need to do so, though things. I guess that in next year talk I'll, probably say: hey we're all good.

A

Everybody is now aligned on the on the w3c, but but it's at this point of time it's it's a bit annoying and also the semantic convention is not always spot-on, which could be a bit annoying.

A

So if I want to kind of sum up what what I'm trying to say is any component that is part of the application, any component that is affecting how the application is running. It's worth monitoring it works to observing. When we look at the race, we think we think this trace um is going to represent what really happened and to do that.

A

We need as much data as as we can get, and I think there is something interesting with service measures, because most of the company that I saw um they use an open temperature gesture implemented in them services, and this is kind of the first time that we see something that is not part of the application code. That is start to emitting telemetry data and I think- um and the panel uh discussed it a bit that we're going to see more and more components start to emit telemetry data as we go on.

A

I don't know, databases and message, brokers and queues and those kind of things they are going to follow it sometime and we're going to see probably the same kind of problem that we see with service measures. So we can kind of look at that as it as a community and learn from that.

A

What we need to do in order to guide those who are going to omit data that we're not going to have those uh same um problems or our mistakes again.

A

So um thank you very much and if you have any questions that I love to to try and answer.

B

Well, thanks for coming.

B

First of all, thank you. I I did the same similar exercise that you did and I came to the same conclusion, but I always been surprised on how kiali is able to create the traces of the box without any hotel implementation. So you have the context propagated, so I've been digging digging again. I have never understood how kali was able to do that. Have you found which.

A

Project I didn't get that kiali, um so I'm not that familiar with that um and it is very interesting to see um and I'll I'll try to dig up and maybe I'll come with a conclusion, because if, if there is like a magic button, you can click and get good traces like with a context. That's a great offering.

B

Anyone else questions.

B

All right well, thank you very much big round of applause.