Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Unified Observability in Kubernetes and Beyond with OpenCensus - Morgan McLean & Jaana Burcu Dogan

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Unified Observability in Kubernetes and Beyond with OpenCensus - Morgan McLean & Jaana Burcu Dogan, Google

Platforms like Kubernetes have allowed developers around the world to develop highly performant and easily configurable distributed systems. However, the techniques that developers have used in the past are no longer as effective in these environments. OpenCensus is designed with this situation in mind: it provides libraries that automatically capture application metrics and distributed traces, attaches the correct service and workload metadata, and sends them to your backend of choice for analysis. In this talk we will: - Instrument a set of microservices, including custom metrics and tie-ins to OpenMetrics - Analyze the telemetry captured from the app with common open-source tools like Prometheus and Jaeger - Demonstrate advanced features like exporting data to multiple backends and changing configuration in realtime without requiring redeployment See opencensus.io for more details.

To learn more: https://sched.co/GrXm

A

Okay, welcome everyone thanks for coming, it's special! This is a late afternoon talk. We are here to talk about open census and our plans for kubernetes. So, first of all, let me introduce myself. I'm Jana I've been in this industry for a long time and I'm pretty much of a really frustrated engineer. I've been working for 15 years, almost pretty much working on networking systems and I went from company to company. Pretty much do the same thing like reinventing the wheel to scale how we develop and how we scale our production.

A

My job just started in really involved building systems, also making them more reliable and more debuggable. So we are all here at this conference for kubernetes, and maybe it's ecosystem tools right, but what why are we actually here? Why did we all get interested in queue Bernice's to begin with, not for the sake of like learning another tool? We are all here because these tools are making our lives.

A

Whether an engineer's dream is to get things done and then you know, go back home and sleep tight and maybe not interrupted because of an on call and if it's, if there's an on-call interruption, you want to be able to know debug things and go back to sleep again right, and this should be attained of the core motivation of everything we build just making people more productive. But this is not really as easy as it sounds, and the main reason, because is our systems- are getting more complicated.

A

Most systems start really small ended up more complexity over time, even if you have a really small size project, a really nicely scoped project, you end up actually growing in terms of complexity, because you're, depending all these new other things around you're, depending on layers of layers of infrastructure, software and hardware, which adds up to complexity.

A

This is what it looks like nowadays, like you. Have all these different layers. I call all these. Like your code frameworks, libraries, the user space and everything underneath is the infrastructure and cloud stack- is becoming more complicated in the recent years. Like you know, there was no kubernetes a couple of years ago. Nobody really knows about the bare metal. To be honest, few people in the world understands the and network stack is already too complicated, and now we have this, like you know, cloud stack and who really understands all this stuff all and it.

A

This is all that for black box for us, but we all need to rely on to get things done right.

A

Unfortunately, now we are in a world where complexity is inevitable because of the growing dependencies, and you know, even if it's not our system, it's just the things that we rely on is complex. Things fail in unexpected ways and in isolation, but you know in the end of the day, we're still in the charge of fixing them understanding them when things go wrong and if we can't do anything to reduce the complexity because it's out of our control, maybe we need better concepts and tools to understand what is going wrong when things are going wrong.

A

So you may think that life is amazing and everything is well contained when you have a single pod, a single container at this level. You know the problems are well contained right, like the boundaries are well set. This is a really nice abstraction and it's a good isolation level. It looks really elegant. Pretty pretty and simple. Several tools already exist actually like understand a singleton. You can depend on easily on the you know, the Linux, perf and diagnostic tools.

A

You know some of these tools, probably how many of you have heard about these tools or using them like TCP dump. You know Nessa yeah, like lots of you, the DNS diagnostic tools, typical. You know perf Linux perf tools, but then what happens? It's never as simple as like having one single container right. You have two of them and when you have two of them, things are getting a little bit more complicated because you have some sort of networking in between you need to talk.

A

I, simplify the situation here by just having like two containers side by side, but imagine a long chain of different containers depending on each other, and the communication among them is not just networking you may have like different higher-level. You know it's components such as message queues. Maybe you have all this different variety of different ways to trigger work, and then the storage comes into play and everybody knows that, like you know, storage is the most nastiest thing. Storage is the main root cause of all the problems, latent supplies or heart failure.

A

Cases and occasionally network, is getting more complicated than you think, because you are going out of your local network and coming back in sometimes you're, going out of your physical location and coming back in, and you know you have a lot balancer in between even internally in your network and load. Balancers are pretty much getting more complicated. If you are going to the external network and load balancers do a lot of magic. The first magic they do is the load, balancing decisions and I'm, not an engineer, smart enough to understand them.

A

The second one is like you know the typical things that, like retries and like reject and request and all sort of stuff, and when I put all these containers in in naked forms, they looked really simple, but they are actually living in actual host machines. So these are like could be rams, BMS running on VMs and there's all this like different.

A

You know, networking, bridging and so on, like networks, like is really complicated, there are endless failure modes when it comes to you know, talking about deploy and containerized systems around I just want to give you a few of them. So you can understand the level of complexity, and all this stuff in the end of the day is just you know, orchestrated hot mess.

A

You know who understands cube root is nt and I sometimes work with people working on the working on kubernetes, and they sometimes are surprised to see why things end up being in a state. It's unexpected things: I've failed in isolation, even people you might expect to have understanding this. The entire stack end to an end, doesn't truly understand what is going on. Are these problems unique to kubernetes? Well, of course, not like these issues have been around for a long time, but I can say that they are really amplified the first one is.

A

The first issue is the low quality is now an issue because our workloads are distributed on a fleet of machines. We can't always really tell like where the actual issue is routed. It right, like which specific machine or VM network and as I, is a huge. You know made of many components and the network stack itself is already too complicated.

A

Scheduling decisions are really complicated and things always fail in isolation, and not a single person really understands and service dependencies are another new real concern, not a new concern, but now we break down our model, it's much easier and it really amplifies the problems of, like you know, having more services, as well as having more failure in terms of service dependencies. So I give you you know if they're right of these failure modes, because in reality you can't really assume that your pot is an isolated artifact.

A

It really depends on all these layers and all these relationships and all of them really matter.

A

Let's take a look at the these layers again, like you know, it's just too much too much to learn many layers all scattered around, and nobody is a true expert on the entire stack, so pinging a server is the hello world of all networking problems and as simple as it looks, I'm really surprised, sometimes like all the issues I'm going through when I'm pinging a server and here I actually have a ping client that is going to like ping.

A

The pong server phone server has been contributed by one of my co-workers already, so my job is they're, just really write a ping client and see what is going on. I will switch to my demo if I can, and so I already did some homework, and this is the simple main function. I have what it does is really. It takes a instrumented, HTTP client and makes an HTTP POST to an endpoint. What it does is really simple it just logs ping makes a request and prints the response.

A

Still, you know the basics of you know the hello world network and example, and I use an instrumented, HTTP client. As you see it's that simple, that, like I imported this package and amusing, the instrumented transport instead of the gold standard, live transport, so I already have a podium. Oh sorry, pinky I move for this and it just basically just grabs the this pink client and runs it. What I'm going to do is like going to deploy this.

A

And I actually have a deployment. Now, if I take a look at my pods I can see pink is running, it's awesome, I already, as I explained, there was pong, so pong is already running in this cluster and now I can take a look at my logs to see if things are working correctly.

A

Let me actually follow this because.

A

Oh, as you can see, that actual started to work and as I was expecting, it's actually doing some some sort of like what I was expecting like printing a ping and then like I'm, getting a pong as a response. If you take a look at our pong server, it just really responds with pong. So it's working as expected, but there's actually one thing that I haven't expected, which is like once a while.

A

It just prints out this like empty string, which is not what I was expecting and it's kind of funny, because the handler really hard code to respond. So there's no reason that, like it's actually like, you know responding with an empty string.

A

So since you cannot I can't really truly understand what is going on with from my logs or what I will do is to switch to some other diagnostic tools in this case, as I as I was telling you I instrumented already with the with a client that can give me some distributed traces, and here what I can do is to understand what it actually happens in the lifetime of that request. In that ping request.

A

What I will do is to click one of these and, as you see this is this is a distributed trace and it explains what happened in the lifetime of that request and it's funny, because I was expecting that like hey, this is a really simple case, I'm just sending an outgoing requests and like it's going to be handled by the pong server, but you can see that there's actually a lot more going on and none of this actually, what is going on here was visible to me without distributed traces.

A

So I can see that, like I'm, sending the request- and it's actually hitting a load balancer rather than directly going to the server itself and then there's a decision made here and you can see that there's actually a new target and the then the load balancer is proxy in the incoming server requests to the to the actual target and making the making the request, and at some point it's actually going to the pong.

A

Server and pong server in this case is responding in like 32 nanoseconds and everything is just being proxied back to my pink lines and at some point, I actually print the response.

A

So all this stuff was really new to me and if you, if you actually take a look at all these, like events, there's more going on in terms of network and I'm doing all this like crazy, like in a connection pooling and like you know, resolving your DNS, which I couldn't see here, but I can see this level of detail by looking at my distributed traces. One thing I haven't seen here is like this is not a failed case.

A

So maybe let me filter by HTTP status 500 to see if I can figure out an error case if I click on this, maybe hide the events. So this actually looks a little bit different. If you remember from the first trace, I was sending one thing and then the load balancer was responding and it was being received in this case.

A

There's again, this load balancer, but as you can see that there is there's other things going on, this proxying thing has failed, which was not visible to me from the logs because logs didn't capture that level of detail.

A

But it says that, like it couldn't dial this particular target because there is no such target around and it tried three times to send a request proxy the request, but it couldn't and then I ended up failing and it responded me back with 500, but the funny thing is like in my logs I was not able to see it because I am a terrible programmer myself now, I realize that going back to my client, I didn't really handle the error here.

A

So the client post is actually not returning me an error, but you know the response is a 500 response, so I should have like could hear something like hey. If response that is third, you know it's different from 200.

A

Just you know log some error, but I failed because I was not really paying attention and my code reviewer was probably not doing a good job. So what happens is I was able to you know: I realized something that I couldn't gather from logs or anything our team rolled out a custom load balancer. It was in our stack, I didn't know about it. I learned it because we had distributed traces. I had this like missing error handling case and was not login.

A

A message about that case at all and I was able to see it and now I can actually fix my code and our load balancers in the end of the day, was doing a poor job. It was trying to go to a you know, target that didn't exist. So it's funny. I can't do anything about this because I'm not the maintainer of the load balancer, but what I can do is to quickly escalate it to the team working on the load balancer. So three things solved by just having this type of visibility.

A

While ago the open source, a set of instrumentation libraries that is used to you know, instrument the traces that you have just seen, and this is now a project contributed by a variety of you know: cloud providers and APM companies. We have this first-class goals of providing. You know, correlation between different signals. Distributes tracing is just one of them, and I want to leave the mic to my coworker Morgan now, so he can explain more details about what we are doing in that space. Thanks.

B

B

So in Yanis demo you saw how she was able to instrument and get traces out of her application with just a few lines of code right, and so what was happening there is open census automatically, has an HTTP handler and go, and if that it handles contacts propagation and also captures traces and spans. So it's a really easy way to instrument your application.

B

As Janna mentioned, my name is Morgan McClane I'm, a product manager at Google I've, been on the open census project since the very beginning about one year ago and cube con Austin, we sort of announced open census to, admittedly a much smaller room than this, so I'm glad glad the community has grown quite a bit since then, we've had a number of other companies, a number of other organizations during the project. We have a lot of contributions and it's it's really really grown and gained a lot of momentum.

B

Some more background, open census is a set of instrumentation libraries that allow you to capture traces so distributed traces, propagate context and capture application, metrics that includes typically automatic HTTP or other RPC metrics, as well as custom metrics. That you can find so. Let's extend Yanis example to a more typical ecommerce application, so here we're looking at about four services, typically any commerce that but for services, but that's what we're gonna look at today and we've drawn up some of the interactions between them just like in our demo.

B

One of the challenges is getting information and getting Salama tree out of these surfaces and, as yanan demonstrated, we can use the open census libraries to do this. We've libraries across seven languages, you link them to your application, rebuild redeploy your app and they'll catarrian capture as much salami tree as possible. If we can't capture it automatically, you can either add instrumentation to the open census, libraries and please put in a pull request to get up, so everyone else can take advantage of it or you can just instrument your own application.

B

Have it used the open census. Api is to send data they're, not super fancy, probably not not. You know as sort of a wild concept. Anyone here there are some limitations here, though, primarily open census, because it's running in process as a library can't capture system metrics. So you can't capture like system, CPU consumption or anything about the hosting run. Secondly, you, if you want to switch exporters, so open census can send metrics and telemetry to various different backends.

B

If you want to change which back-end you're using you'll have to recompile and redeploy your app, which is really really annoying, it's especially annoying if you are providing a platform like kubernetes itself, right like if you're a cloud provider or even a company that uses kubernetes if you're switching export targets, you probably don't want to have to like rebuild all of kubernetes, that's sort of not desirable, and to that end, we've created the open census agent. So this is an optional component.

B

You can still export things directly from the open census, libraries, but what it does is it runs on every VM or every kubernetes pod, or really every host that you have and you can have the open census. Libraries sent ulema tree to the open census agent, which can then send them to your metrics or tracing backends. The advantage of this is that you can change exporters on the fly without rebuilding or redeploying your app. It also means that it can collect system Salama tree particularly system metrics, and send those through that same back-end.

B

So it's a huge extension of the existing open census ecosystem and, as I mentioned, this is completely optional. This is in beta. We I think we have a few customers using it. It's on github, it's definitely usable, but there's more work coming the system, metrics part, isn't there quite yet, and finally, we have the open census service. We're about to do a demonstration of this on stage. The open census is another service is another optional component. You can run it as a service inside of kubernetes really anywhere else. You deploy container.

B

It runs as a network service, and you can send your traces and metrics and telemetry to the service, which will then send them back to your back-end of choice. So it's just another layer of aggregation, but it also brings in some of its own really powerful features. The first is tail baselet trace sampling. So typically, when you sample traces in application, you need to sample them, make your sampling decision at the beginning.

B

This allows you to make the sampling decision after the traces are captured, so you can get much more interesting traces sent to your backends. This is like incredibly incredibly powerful. Next, you can change your exporters more easily. You can do it fleet wide and finally, open census includes a feature called Z pages where you can sort of get this firehose view of everything. That's happening right now. This lets you do it across your entire cluster, which frankly makes them actually really useful.

B

This service is in alpha, so it's also on github, but it's probably not ready for primetime. Quite yet, but we're gonna do a Dem demo it anyway and we think will be production ready pretty soon. So I'd like to introduce Steve from omniscient to do the demo.

C

Thanks Morgan, so for those of you who don't know me, my name is Steve Flanders I am head of product and experience that is still startup that you probably haven't heard of called omniscient and, while I'm not going to talk too much about what we're doing today, what I can say is that we are contributors and maintainer zuv, the open census repository with Google and others.

C

So, just to summarize really quickly what Morgan was talking about, where the open census service, it's made up of two different components: the agents and the collector and there's a variety of different use cases for why this service exists. Now these two components live with your application, and you can send to a variety of different backends, there's a variety of different reasons. Why you want to do this? The biggest is when you're, adding instrumentation to your application.

C

You want to do it once you want to kind of set it and forget it, and so you can do that with the open census service, because you can lazily configure anything that you want in terms of exporters, sampling policies and more and that's what the open census service is trying to solve here. It does so by having two primary components: receivers and exporters, as the names imply receivers. Allow you to get data into the system. Exporters are a way of getting data out of the system.

C

Alright, so there's a healthy ecosystem of options here for receivers and exporters for both traces and metrics. Here are just some of the that are available today in the open census, repo and, of course you can contribute your own as well. The goal is to provide flexibility with a standard in place. All you have to do is write a single exporter and you can then leverage the power of open census.

C

Now. What I want to talk to you about is something in the open census collector. Specifically it's the problem of traces, which is they're extremely verbose. There are a lot of them, they're extremely valuable, but there's a lot of duplication. For example, you probably want to see some of the HTTP status 200 messages that are coming into your system that are being traced, but do you need to see all of them? Well, I mean if you can compute things like latency, and you can only get traces like errors that you care about.

C

You don't need to get every single events, that's where sampling comes into place. The primary way of handling sampling today is something known as head based sampling. This happens either at the client or at the agent level. It's a sampling decision that, when the trace is being instantiated, when the first span is being created, a decision is made and whether or not we're going to actually sample that event and send it to our back-end of choice. This is great, it's extremely powerful.

C

It allows us to limit the data that's coming into the system, but it has some limitations because we're making that sampling decision when the trace is being instantiated. We don't know what actually happens downstream in that call. For example, if the next span, the next service, that I call has an error, I may not receive that error, because I made a sampling decision. That said, drop that event so to solve this problem, there's something on his tail based sampling.

C

The idea here is I actually collect the entire trace and all the spans in it, and then I make a sampling decision. This is powerful because I can have a variety of different policies that dictate whether or not I want to sample this event. A very classic example is the error based. So if a trace has an error, maybe I want to sample a hundred percent of those, whereas if it's not an error, I can use a different head based sampling policy, but that's really just the tip of the iceberg.

C

You can actually do some much more interesting things as well. For example, these retrieval based sampling policy. The idea here is, maybe I decided not to sample this I can still keep that trace in the collector and I can ask for it later, maybe a minute later, ten minutes later an hour later, so the trace isn't actually gone. I chose not to sample it, but if I then realize I need it, I can still get it back out of the system.

C

Okay, so I'm actually going to demo this. This is a feature. That's gonna, be upstream to the open census service repo. Here in the coming weeks, we have right now as a prototype that I can show you live. So in the open census service we use something called the hipster shop, it's a demo application and it has a topology. That's apology could then be replicated in some sort of load generation tool, so we've written a load generation tool, it's JSON configuration based and right now at Mayer, so that mirrors the hipster shop.

C

But you can actually write your own configuration your own topology for your environment as well. Then what we've done is we've pointed this log generation tool at the open census, collector the open census collector receives it, and it's been configured to send it to different destinations, something going to be running this locally I'm, sending to two separate Jaeger instances, and you can of course try this out to everything I'm about to show. You is publicly available right now, it'll be up streamed and open census repo in the coming weeks.

C

So if you haven't seen the hipster demo app, it's basically a web application, it allows you to it's basically a marketplace where you can order some cool hipster items. It's actually a polyglot micro services based distributed architecture, it's written in a variety of different languages. This is also available on github. You can play with it it's a way of actually sampling all of open census capabilities.

C

So what we've done is we've taken that topology on the right hand, side there and we've created a configuration for it, and so what I'd like to do is go ahead and demo that, for you, okay, so that topology, that you saw on the right hand, side, here's a more live representation of it. So you can still see all the services. The error is here indicate the amount of traffic, the amount of requests per second that are being sent and anything with red indicates errors in the system.

C

So this is synthetic data that the load generator is sending to the open census collector, and what I want to do is make a sampling decision based on the types of traces that I'm receiving. So let's go ahead and do that in the demo application it's a little bit small, it's trying to make it a little bit bigger for you.

C

We have a docker compose file and this will actually spin up a Cassandra instance with two different key spaces, because we've deployed two different Jaeger instances, and so what we've done is we've configured the load generator to send to the open census service. The open census service has two rules, one to send all events to a single jäger instance, and one to only send error traces to the second one. Let's go ahead and take a look at what that looks like here.

C

As this system comes up, so it takes about ten seconds or so, and then you can see right now. It's emitting trace IDs. This is going to actually represent that visualization. That I just showed you, let's pull it up, so it actually exposes two ports. It exposes 17000, which is a Yaeger instance, that's receiving all of them. So, for example, I'm looking at the checkout service here, which is one of the hipster store applications.

C

You can see here that I have some traces that have no errors in them and I have some traces that do have errors in them. So this is the Yaeger instance that the open census collector is sending a hundred percent of the traces to today. Basically, no sampling policy in place great there's, also a second one running on port 18,000. This one also has to checkout service but notice that there are no success.

C

Events there are only errors, that's because there's a tale based sampling policy in place that says only sample the error events and drop everything else. This is the power of tale based sampling in just a single example. There are several other policies that can be written here so just at a high level to kind of show how this works. The actual topology generation is JSON configuration base. You can see an example of it here.

C

You can write your own and then, on the flip side of that there is Gamal based configuration for the open census service. So here you can see that there are two policy rules, there's one where we're actually doing it, based on tags, another one where we're doing it on HTTP status, codes, errors, for example- and this can be extended to a variety of other use cases as I was highlighting before so.

C

With that I'll turn back and everything that was just demoed will be up and the open census repo here in the coming weeks, and with that I'll turn it back over to Morgan to wrap it up.

B

Thanks Steve, that's fantastic, so I want to wrap up today by talking about three other important tasks that we're working on. So, in addition, everything that we've shown you this is only you know, a thirty five minute presentation, there's some other there's some other items that we want to talk about. The first is there's a lot of questions that we've received throughout the life of this project.

B

About what's the difference between open census is tracing API and open tracing we've been meeting pretty regularly with the open tracing group to current try and come up with a common API. We actually at a meeting I think two days ago and early in the morning and we're in a pretty good track. We're really excited of the progress we've made and hopefully we'll have more to share pretty soon in the in the new year. Next in 2019, we're also adding client-side instrumentation open census.

B

So today, open census, like many other tracers, focuses on tracing your back-end systems. Things running inside of a cloud provider things running inside of your data center.

B

What we're adding and there's already a prototype out today is instrumentation for front-end web applications, so you can actually, you can get spans in your traces that are emanated from a client, apps apps, and so you can see like client, latency and the amount of time it takes for a lot of UI events in your clients later in 2019, probably late 2019 we'll be adding support for Android and iOS as well.

B

Finally, we're in the middle of a design process for open census, logging I, don't have a lot of time to talk about this today, we've already met with a lot of larger, open census, users and vendors, but if logging and and sort of new logging, api's and new logging technologies or something you're interested in, please stay behind and chat with us, and with that we're done. So. Thank you very much for attending. Today. We really appreciate it.

B

Our goal is always being to make this project as open as possible, and we have a really big and growing community there's. A number of us here from Google Microsoft is one of our biggest partners. Sergey is here for Microsoft, omniscient and other partners to stay here and chat, answer questions, and just hang up so with that. Thank you very much.