Open Telemetry Uncategorized, 16 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OPS Live w/Suman Karumuri

Description

Austin Parker and Liz Fong-Jones interview Suman Karumuri of Slack at OPS Live.

A

So Suman is an old friend of honeycombs, um so I've been getting to a little.

B

A

um So thank you for coming here on to the show zoom on. Thank.

C

You thank you for having me um so.

A

Zoom on just gave a critically acclaimed talk at observing practitioner summit. Yes,.

B

And just to reintroduce the segment I'm Austan Park, with a light step.

A

Honeycomb- and this is c1 from stock.

B

Yes, so how's it going good.

C

Yeah excited to be here: yeah yeah.

B

Pulling just a little bit to the mic: okay, yeah cool! So can you tell us a little bit about your talk? Yeah.

C

So our talk is, we have started doing distributed tracing and.

B

C

So like we had prior experiences of how tracing went in other companies and we wanted to avoid some of those pitfalls. Okay,.

B

C

Have kind of rethought about the pitfalls of tracing and then built in your system.

B

Andressa, so what would the you like, elaborate on the pitfalls? A little bit yeah.

C

So one of the pitfalls is the biggest way I think about like the biggest pitfall of tracing systems. Today is the tracing systems that we have today that you can capture the raw trace data mm-hmm.

A

But pranic each span is a structured event, yeah easier.

C

And is a structured event precisely yep, and we want to so. The vision for us was to break that silo. It's like in the current systems. You get these fixed views into the data, so.

B

C

You it's basically like giving you all this crazy data and awesome data like awesome, insights and.

B

C

You a fixed like a single dashboard for your metrics data log data, so we asked hey like what happens if you can access raised it as raw data? What changes need to happen right.

A

And it's so powerful because you only have to store the date at once right and it's kind of slicing it along.

B

Though um yeah.

A

My inner sense.

C

Just a little bit, we have other talks where we presented mirror on to a broader audience, but I just said it's like a scribe. Like a.

B

Lot with time durability is like you know, the tools are cool and alright, but what sort of changes have you seen camp organizationally you're like? How is it helping your team's out? Can you talk about that? A little bit yeah.

C

So I think observability is becoming more critical, mm-hmm in running our systems and making them reliable and also the other thing I feel observability going on is I mean historically, it's viewed as a cost center, where it's like yeah.

A

C

If it's up its up, it's.

B

C

Data, but as people rely on having good, we can deliver consistent performance for.

B

These customers.

C

And having the pulse of your system in real time, all.

B

Right and then.

C

Acting on it to make your customer experience better, more proactively, okay,.

B

C

Think has it is changing and absorb. It is becoming more and more important in.

B

Your means definitely definitely so.

A

What are kind of some of the things that you're using observability for to to solve it flock so.

C

I'm gonna be just like any other company we have our matrix pipeline. A logging pipeline and a tracing pipeline and in the matrix pipeline is mostly from me, dashboards and alerts for such as this really great, and we use the trace data in honeycomb and our data warehouse and to get more deeper insights into events that are correlated like. We can ask hey ma'am like how much does the p99 of memcache affect our customer experience of slack, and we can prioritize issues based on that. We can prioritize what to fix when to fix and how.

B

C

It affects based on that cool.

B

A

It's super great to kind of supercharge your ops teams right and supercharge your customer, so success teams right when you give people access to the right data right.

C

A

Kind of decrease those organizational boundaries right right because of you know, hey the customers. Success agents know how to use the data warehouse right like it means that everyone gets access to the data they need to confirm. Like you know, is you know a fortune, ten company having problems, yeah yeah, exactly.

A

So what sorts of questions you get after talk? I heard two rounds of applause, so I assume. That means that you know you finish your talk and people ask questions. Men, yeah.

C

I think so so I think I ought very interesting questions. Some of the questions were kind of like holes. In my talk where I kind of like glossed over some of the details that I assumed people would already know, but yeah.

A

C

Because of the wide variability of Stax, it's not like some things that people really.

A

Hate for granted, you know when you work at Aflac or a Google right. Like assume you know, you'll just have a service message. Just works yeah.

C

Yeah, so so there's some something like that. The other thing interesting for us was people who are surprised that sampling works and.

A

I was having to explain this over and over and over rings.

C

A

You do metrics, metrics are the ultimate form of sanding.

B

Right, it's like.

A

When you're sampling right think you're getting the metrics data, just with additional exemplars like high cardinality, feels.

C

A

A metric is right.

C

Yes, matrix are basically pre, happy edges, Santos yeah.

B

I think the reason people are afraid of it is not like it's not because I mean sampling is good. In some cases. It's bad in some cases rightly.

A

Like you know, just straight probability, sampling can be really lost, but.

B

A

You know honeycomb and white step and so many other people care about making. They may look sampling really work. Yes,.

B

A

That, when you can not have your you know, a hundred thousand QPS services drown out your thousand PPS services. It.

B

A

You can actually see everything yeah, proportional volume and.

B

You need some architecture that can support that right, like there's different ways to do it. There's, obviously, you know open telemetry I think has a really cool. It is pushing the state of the art forward a lot here because of the collector and it's a supporting stale based sampling. So this is becomes more accessible. You know in an open source level, but you know there's a lot of I feel like that's gonna, be the next big challenge of observability yeah.

A

B

A community is like kind of taking sampling back.

A

Like you know, large Kayode use and people like.

B

A

On saying like yes, we have done this. Yes, we have access to the right data. No sampling is not you know. No sampling is not like. You know like losing you fidelity. This is actually essential to actually being able to measure your whole system right.

C

B

C

Picture sense, yeah. The other thing is also how we do sampling, because we always do guide based sampling, yeah.

A

C

When I said head based, sampling I just assumed they would do I mean that was obvious, but a lot of people were asking me. That was the other question like do two head based or tail based. We do head base sampling because.

A

B

C

There is a big problem with trail by tracing overhead is not insignificant. Yeah.

A

It is significant to do performance, regressions, yeah, yeah, I'm.

C

More than performance equations, one of the things we noticed is let's say: I am NOT, giving any concrete numbers. But let's say the overhead is 10%, which means you need to run ten percent more infrastructure, yeah.

A

C

To get these traces and.

A

C

Of the problems we saw with, that is ten percent at our infrastructure level. Humans are cheaper right.

B

So but I would also look like caution. People to like this is slack. We're talking about right, I think a lot of people. You know, there's a very large spectrum of companies that have observability challenges that are like somewhere between, like slack and one person like Raj and yeah,.

C

I need to be a you, have a system that can sort of gradually scale yeah. It is a trade-off, so I would say like at some point. The human time is more valuable like initially, if you had a small-scale, the human time is more valuable, like see like tracing everything and like let humans look at it. It is very valuable, whereas.

B

C

At larger scales, humans tend to be cheaper. Sometimes the.

B

Infrastructure, the.

C

Other thing we actually augment that is instead of doing black and white sampling, we.

B

C

Have tired sampling? We.

B

C

1% base sampling, where we trace all the requests, yeah.

B

C

We actually select for specific use cases. We turn on sampling for the specific use cases yeah.

A

That was one of my favorite things about Darwin I was like.

C

Yeah when, whenever.

A

In trouble, I could just say you know what okay, like the.

B

One for one like one minute.

C

Right yeah see what's.

A

Going on, it's almost like getting attaching a debugger or profiler right exactly.

C

Exactly and everything.

A

Is right like head based sampling does not have to mean not dynamic right, you can do head based sampling on properties. You know, and you receive the request right. You.

C

A

Your customer ID, it is right, like.

C

You don't have.

A

To detail based sampling to get that right.

B

C

A

We conflate, you know, table sampling and dynamic and I, don't think. That's true right, I think.

C

A

Clearly doing these terms, which I love, this conversation is getting there because you know a while back, it was like you know: just people were exhausted and just terminology I.

B

Think they still are in a lot of ways: I think that, like sampling, is still sort of one of those. It's a technical term that entered into the marketing realm and is still sort of hanging out there, because.

C

B

Of people differentiating it like there's a lot of people with products are differentiating on it. Right, like I, mean I would suggest that, like both of the companies representatives stable, like differentiate their products from the rest of the market in large part because of sampling and I, would tend to agree, dynamic sampling is a much better liked. um Yeah, you know yeah I,.

A

Know- and it's the only way that it can really be cost competitive, metrics right.

B

Otherwise, you're.

A

You're stuck with collecting not enough traces or collecting anything.

B

About right, yeah, just I mean if the other day there's a cost. You know, there's a trade-off everything there's a cost for everything, but I would suggest the one thing that I do kind of tell people like hey. You know when you go and you see these talks and you look at like the implementation of large skillet or tracing systems at Google at SLAC, at Facebook and whatever, like decisions they're making are incredibly informed by the scale they operated and you probably daren't operating at that scale. Right now, in a lot of cases, yeah.

A

Yeah but the more that we get things are kind of into the public space from large companies right then it benefits everyone.

B

Yes, right absolutely kind.

A

Of decide which things are appropriate for your scale and when you decide to scale up there will be a pretty.

B

Something ready implementation that you can use right and that's why I like the dynamic sampling is sort of approach to this, because it means that when you're at a smaller scale- or you have slower cert, you know smaller surfaces, slower surfaces or whatever like oh, you can just turn it up to 100, then and then, as it kind of moves up like yeah. Well, there's some heuristics. This is math yeah.

A

One thing I'm curious about is kind of in the past couple of months, since I last saw this like how is the developer experience of telemetry and instrumentation evolved? That's like yeah.

C

So I think it it is actually I mean, as you know, it was very widely well received. I think the biggest transition for people is. There is a little bit of like abstractness to it like if you know tracing you know what it is useful for it's good, but until you see your actual application, traces like that turns the lightbulb on for.

A

The right people mean that aha moment of okay, now yeah.

C

Yeah yeah and wants to get the aha moment you just can't stop them, but getting them there is the hard part. I think I've done this with several companies serving teams and get getting the team to that. Aha moment is like the most important bit and the other thing is: we have to make the observability pipeline whether it's metric, logs or traces super smooth like it should be easy to onboard. The right thing should be the easiest thing to do.

C

Yeah and once you can get that pipeline very easy to on if it's like, if it works like magic, that's it yeah.

A

That was kind of the friction that I was seeing when I was working with one of your teams. A couple months ago of people were like. Oh my goodness, right like this API is like super yeah.

A

You know I. Imagine it's been a couple months now.

C

Yes much better yeah now it's like we have built so the Burin API was a low-level piece. We have abstracted that to the applications. Now. Oh.

A

That's good yeah because have yeah well directly send event yeah I like that means they have to worry about all their tracing yeah.

C

What do they I'm actually done? Is we built this arc? I mean there is a slack architecture diagram where you have a HTTP endpoint, where people can find traces to and.

A

B

C

Thing we did is we built a command-line client which, which is a program- that's installed in slack base, so it's everywhere and you call the command client with, and it's just like an exec. So.

B

C

Can use it to send us pants and.

B

C

This guy this /, it's called neuron tracer that writes the spans to HTTP endpoint and then all the abstraction, all the internal complexities are gone and the all the users see is okay. What race ID do I said what tags do I set and.

B

C

Then, if you pick, we pick reasonable default, so it's super easy and now the adoption just that little thing made an option. Very easy. Yeah.

A

That's the same thing that we've found with other folks is that when you implement distributed tracing on a CI infrastructure right, yeah.

B

C

Know in AC.

A

Value out of it, because you know one of the number one frustrations people know that they have is: oh, my god, I'm tearing out my hair wire ride belts as well. Yeah.

C

Yeah actually, the other reason tracing just works for CI I've seen this clique multiple times, the reason being, if you think about it, CI people are deal with graphs like your build tree, is a dependency graph. Yeah.

B

Yeah Jenkins job is a dependency graph.

C

So it is a very natural fit for them, so they'll immediately see the value in systems. Ladies.

B

I think it's also one thing: I've talked to some people about is taking tracing and putting it on the CI side, but more like looking at infrastructure, automation, right.

C

B

If you've used ansible before.

C

Yeah even like the most.

B

Unobservable programs and it exists.

C

Oh, my god, yeah.

A

C

You trying to understand what's happening, yeah chef.

B

Cookbook is running yeah.

C

Yeah, we actually are instrumenting right now, like the chef events with instrumentalist.

B

A

Which is brilliant, yeah yeah because, like we need this like for for chef like okay,.

C

So you've been having.

A

These super annoying problems where honeycomb, unfortunately, our ingest jobs have been failing to spin up fast enough, like it's taking ten minutes.

C

Oh yeah, precisely the problem we are trying to solve, like I've, probably takes like 30 minutes. Sometimes it's.

A

Like we have no idea why it's like there's no way you can run responsive enough service if your auto scaling Peaks 30 minutes right. Yes, yeah he's a scan, yeah and.

C

By the time you scale up, you have probably yeah.

A

It's two-way airplanes. These are cratering like.

C

The stuff is happening, yeah.

A

Like this, this is a super Richfield. Where you know you can see me operational benefits and it kind of doesn't require necessarily that much instrumentation of application-specific fields right eventually, you get people the point where they, you know right: application-specific structured logs in the form of key belly hairs, but you know it takes a while for them to get there. Yeah.

C

B

Yep well, I definitely hope that we see more. You know more in that direction and I. Think open. Telemetry is a good step towards this right, where it resolves a lot of the sort of I.

B

Don't know complete, no I've got complexity, but the confusion right like saying like okay, here's one data format, one header propagation, you know one way to propagate context, well-supported libraries in SDKs in every language, and then you can just write an exporter to whatever your same thing.

C

B

So if you've got your own kind of pipeline or you need to engine, you know.

A

Honestly, I think it's probably the opposite in for software they've written all of this in-house libraries right and I think that you wind up wanting to say you know what happens if one, if one day they decide that they want to use some other provider. In addition, honeycomb right now.

B

Yeah, you kind.

A

Of have to make sure that you're outputting in a standard format and everything else consumes that form and then it makes the switching easier.

B

I think that and I feel, like that's sort of the next step for a lot of there's a lot of people. You know there's a lot of companies that are talking about. Like oh yeah, we you know we're supporting up until imagery and we're donating time and resources. Engineering, wise, I, think the real interesting thing will be like. Okay, when does everyone kind of allow you to ingest open till Emma tree data format, natively yeah,.

A

And that's where a lot of the protocol work that be formally a mission now, it's Munk folks are doing where they're doing the collector work right, such that you know one day like it shouldn't be necessary to have everyone running their own, even their own, specialized in DES format. Right like we should all be.

C

A

Open telemetry format, yeah.

B

A

Let me see whether folks on.

C

A

Have any questions for you, sumon yeah,.

C

A

People to ask, and one of the things that I have found is unfortunately the Wi-Fi here requires reconnecting every hour, so I keep on getting kicked off the Wi-Fi.

C

I don't know, I was trying to do a live demo and I. Couldn't do it because you.

B

Know it didn't work I.

C

Didn't know how to reconnect to.

B

The it looks like yeah.

A

It's going a little sour, but.

B

I finally found the local statistics on the air for some reason. There's this fun like distribution on the CUSO rtmp queue like.

C

A

B

I think that's why it keeps complain about stream quality. Yes,.

A

Well, you know we're doing the best we can yeah.

B

A

That's really the best that we can hope for. Yes,.

B

Unfortunately, we have very little control over the internet quality here, yeah.

A

Unless we wanted to pay, like you know, hundreds of dollars for a dedicated, Wi-Fi hotspot, which we you know we could like, you could just take like a Verizon yeah.

B

Yeah, maybe I should have done yeah. That's.

A

B

For next time, you.

A

Know yeah testing on production, yeah.

B

Hey this is trust me. This whole thing has been testing the fraud that we had a I think the best testing a problem that was earlier only went five minutes into a site interview, and then someone was like hey. You left the music.

B

It's like oh well, sounds like.

B

Would that be other BJ? Oh wait. Okay,.

A

That maybe James starting yeah, we motive finished.

A

Well, thank you much so much for join.

B

Us in my thank you, yeah.

A

Thank you. It was good having you yeah and we look forward to hearing more from sac, probably.

B

Like more blogs,.

A

And so forth, sure yeah awesome awesome.

B

Alright, thank you. So much we're gonna take a short break and we will be right back.