GitHub Universe 2017 - Featured Sessions, 25 Oct 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Fireside Chat with Charity Majors - GitHub Universe 2017

Description

Join us for a spirited conversation with Charity Majors from Honeycomb.io and Sam Lambert from GitHub about trends in software development. Charity Majors brings her battle stories and experience in metrics and monitoring from Parse, Facebook, and Linden Lab to discuss why she created a company focused on observability for a distributed world.

About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.

For more information on GitHub Universe, check the website:
https://githubuniverse.com

A

♪ (jazz music) ♪.

A

Hug accomplished.

B

How's it going? it's going, well. okay,. So, let's start with introductions.

B

Let's have a little bit about what you're doing now, where you are, and what brings you here, today.

A

Well,, I'm here because ashley asked me: to...

A

(chuckles) honestly..., I'm co-founder of a little startup called honeycomb.

A

I've been in operations engineering since I was...

A

I think I've been on call since I was 17.

A

worked at linden, lab on second life during the crazy days when we thought that vr was going to be taking over the world..

A

I think I took it from 50 servers to over 8,000 in four different co-facilities.

A

Couple other little startups.

A

parse, I joined in 2013.

A

Was doing back-end engineering there through the acquisition by facebook.

A

And from pre-beta to over a million mobile apps...

A

(Sam lambert) wow. ...by the time I left, yeah.

B

Wow, so you're running honeycomb now..

B

Do you want to tell us a little bit about the business?

B

(charity) sure, yeah.

A

Honeycomb is a company born out of rage.

A

I really hate--, I hate monitoring.. I hate everything about it.

A

But it's increasingly important and in ways that are different.

A

I know we'll talk about that more., but basically it's an explorable way. Of...

A

Observability, basically, for software engineers.

A

It's for more complex, distributed systems than what we're used to in the past. (sam) okay.. It's very distracting;. I can see my head up right above you.

A

yeah.. I just thought I couldn't look at that. (chuckles).

B

So you've been on this journey throughout your career through various different companies and when I sit down and speak.

B

To founders of their own companies,, they usually tell me that they are--.

B

They've seen something that's missing in the industry.

B

They wanted to see, come about and come into fruition.

B

What did you feel that was missing that made? You want to start honeycomb?.

A

My sanity... (chuckles), so at the point when we were at facebook.

A

You know, they're trying to get us to adopt all these tools at parse we had plenty of problems, right?. One of the biggest problems was that over half of our development time.

A

Was going to searching down like this incredibly long.

A

Infinitely long fintail of things that almost never happen, except for once or twice they did., which doesn't really lend itself to the model where you're curating, your alerts,, things are breaking predictably.

A

And it was taking up all of our time.

A

and we ended up getting ourselves into scuba, which was the one tool that facebook gave us that changed our lives, because we went from hours or days tracking down these insanely complicated problems..

A

That were like an intersection of five rare things that has to happen,, or it only applies to this version of ios from these countries,.

A

Using these language packs and on this firmware, version.

A

You can't monitor for those things.

A

When I was at facebook, I heard so many engineers saying "the thing: I'm going to miss most about facebook is scuba.".

A

I said it so many times. I'd never planned on starting a company., but I felt like using the tools that we built at parse and scuba.

A

We became 10x engineers, to coin a phrase, and I didn't want to live without it,, you know?, and so when we left, like that's as far as I really got.

A

It was just,, people were like "yeah,, you should start a company,, here's some money." and I'm like "okay, well I'll, build this tool,, we'll fail, I'll open source. It.

A

And I'll just never have to live without it.", that was the grand vision., but once we got to it-- (chuckles) yes., once you got a ways into it, I started to realize that more and more people are having these categories of problems..

A

It used to be that the googles and facebooks of the world were here and the rest of us were over there and the more time goes on the more it's like "well, no,. We kind of have the same problems. Sooner.".

A

And the problems of complexity are just becoming everyone's problems.

B

(Sam) yeah,, so when you talk,, you often speak about observability.

B

Versus like metrics... (charity), yeah., ...and monitoring., how would you, define the-- you know, there's been some a+ snark.

A

On the internet, about observability lately, being just the coolest rebranding of monitoring, which--.

A

Excellent snark. a+.

A

it is a different thing, though.

A

or it should be.. When I use it,, I mean observability the way the term--.

A

The terms comes from control, theory, right?

A

and it's just,. It's a superset of all of your instrumentation,.

A

All of your monitoring,. All of the questions that you need to ask of your system.

A

Observability and control theory means basically how well can you understand the inside of your system by looking at the outputs of your system, right?, and if you instrumented it really well,, you could ask arbitrary questions.

A

That you don't have to predict,, which is important, because...

A

Our entire generation of tools was built to answer known unknowns, really fast and really efficiently..

A

They do that. (sam) yup., but increasingly the problems that we have,. It's like,.

A

"Well,, I'm getting a half dozen reports., probably some of them. People just didn't turn on the wi-fi or they're wrong." users are often wrong., but I sense that there's something here..

A

If I knew what the fucking question was, I could answer it, but I don't know what the question is, you know. and this is eating up entire teams' worth of head space.

B

(Sam) so we found throughout github's growth...

B

It's exactly the same problem.

B

we were able to preconceive of issues and problems that we get in our infrastructure,.

B

And they were often never really the problems.

B

Like the alerts that fired were symptoms of another problem,.

B

That was harder to find then we'd kind of be feeling through the day.

B

They're actually trying to find the source of the noise.

B

Or the actual, the root cause of the problem., it turns out, if you're a good team you're, not just going.

A

"Oh, that again. oh, that again.", if you know what the problem is, you fix it. yeah. what you are left with?.

B

Yeah, and we always found dashboards, were the result of an incident,.

B

Rather than something--, there was never a dashboard that was just going to magically represent what was going to happen.. They always came out of learning about a new failure. Case.

A

(Charity) totally. every dashboard is an answer, right?.

A

It's an answer that you can jump to., and this has worked really well for a long time.. Every postmortem you've ever been in probably ended with some version of "what dashboard. Can we create so that we find this problem instantly next time?", right?

A

and, as time goes on, you get these amazing eco-systems.

A

Where you have dashboards to find your dashboards with your dashboards of your dashboards. and half of them are stale. but they're. All answers, right?

A

and, like the metaphor that you brought up, was it's like going to the doctor, and they're like,. "You have scoliosis.".

A

(Chuckles) "nope,, you have a broken arm. you're depressed.", it's like answer,, answer,, answer, answer.. It's not actually-- you're, not actually exploring. you're, not actually engineering, you're, asking questions, or interrogating your system.. Yeah, absolutely.

B

So these world of stuff--github.com is a large monolithic application.

B

And we're exploring a journey towards service orientation,.

B

Splitting things out for organizational and technical scalability.

B

What's the role of observability in this kind of service-oriented world?.

A

Yeah, well,, when you look at the major trends in software development that we see out there, you've got,, serverless-- terrible term but kind of a cool...

A

(Sam) we'll get to-- (chuckles) or micro services. to me, they're symptomatic of the tipping point that we've come to.

A

When it comes to complexity. now I hate it when people toss around the term complexity as much as anyone does--, it's really annoying,. It's not specific.

A

quantify it., but there is a real thing. That's happening where we can no longer do the things.

A

That we want to do with the tools that we have.

A

and we've just kind of gone, "all in,, alright, cool.

A

Let's do micro, services." which..., we were doing micro services back at linden, lab,.

A

Ten years ago! didn't have a name., doesn't matter. you get to a certain point of complexity,. You have to spin things out.

A

So that you can breeze in about them, right?

A

now, when you've added that network cop in there,, you could no longer attach a debugger, right?

A

and I feel like what we're doing,. What a lot of people are doing is where we're all kind of trying to inch toward this holy grail.

A

Which is a distributed gdb or distributed strace.

A

And the other critical thing that's happening there is that.

A

We're getting to a place where the health of the system doesn't and shouldn't matter. right?. The health of the event is what matters.

A

the health of the event,, the health of every user's experience of the events,, the health of every application,, the health of every shopping cart,, the health of--.

A

All of these very high cardinality fields--.

A

Which, again, is something that metrics and dashboards...

A

By virtue of having been stored on disk,, they cannot do.

A

They're, falling apart. dashboards are great at telling you what the health of the system looks. Like.

A

But would you rather have your system be like 25%, down,.

A

Say one availability zone goes down,, but users don't notice.

A

Or would you rather have [inaudible] of people complaining all the time.

A

Right? well,, it's obvious. yeah.. We care about the health of the event. and that specificity, that raw data is what I see observability being.

A

(Sam), so you see the industry turning towards tracking.

B

Individual requests through an entire--.

A

(Charity), you have to. if you're looking for needles in haystacks, every request is a needle and you have to be able to find any one of them by any characteristic.

A

That any of them has.

B

Absolutely., a lot of this sort of discussion around micro services is happening.

B

In hand with the discussion around containers and containerization.

B

(Charity) yeah., what do you feel is most interesting about the movement towards containerizing applications?.

A

How early people jumped on that bandwagon before it was even remotely ready for production.

A

A

Sorry., I'm throwing some shade in containers.

A

honestly, a lot of people adopted containers before they were ready, or before the tech was ready--, but that's great.. That's how things get mature..

A

I think it's pretty clear to most of us now.

A

That the future of software is in containers, talking to services.

A

That are abstracted by apis, and that includes databases.

A

yeah. and that's really exciting. To me.. I feel like there are... back in 2012 when I was building parse from the ground. Up,.

A

It was pretty clear to me: there were some best practices, some ways to do. Things. you use aws;, you choose boring technology where you can.

A

All of these things... and then when we were building honeycomb, a year and a half ago,, I went,.

A

"Holy shit, what happened?" like there was no trustworthy, narrator.

A

There was no conventional wisdom that anyone trusted.

A

And I'm glad that things are kind of starting to converge.

A

On some best practices, again. (sam), yeah,, absolutely.

B

So it felt like four or five years ago, everyone was talking.

B

About moving to the cloud, packaging, their workloads up and somehow getting it--, how do you feel?? Do you feel it's different this time with this type of movement,?

B

The way people are talking, these days?

B

well,? I think that, what's different is...

A

People aren't saying this explicitly as often as I'd like,, but I think.

A

That a lot of what's at the root of it, is that the center of gravity.

A

Is moving towards software engineers, software engineers, owning infrastructure.

A

and a lot of people will get really indignant. When I say this., I'm from operations., I want us to evolve and not--.

A

I don't want what happened to qa, to happen to operations.

A

But it's super clear: that...

A

operations exists on the other side of an api, for the most part, in the future.. It already does for me and I'm in operations, right?

A

most of my ops people work at aws or fastly or github, right?.

A

These big providers, they're doing everything better than you could.

A

And what you should be working on is your core business value.

A

And pretty much nothing else,, because engineering cycles are extremely rare.

B

(Sam) yeah, absolutely. very scarce, so there are clear technical benefits to these types of segmentation,.

B

And breaking things apart, separating the concerns they have.

B

What do you think the organizational effect of making it--.

A

(Charity) oh, conway's, law. (chuckles).

B

Yeah, so like-- yeah. and now, yeah, you're making it.

B

things are no longer within the same codebase,, but now the technology's, more disparate,.

B

And so can the teams be as well behind in these technologies?.

A

Yeah, and this can be really isolating and challenging for teams.

A

I think that in the rush to adopt micro services and do it well.

A

A lot of teams kind of found themselves in a place where only one or two people.

A

Knew how to do anything and they were spread really thinly., and this is a really depressing place to be in,. You know?.

A

At honeycomb, we try to have a policy,, even as young as we are--.

A

Three people know everything.

B

We only have nine engineers,, eight or nine engineers.

A

And I think that that's something we have to be conscious, of.

A

kind of bunching up like-minded things.

A

And co-owning them. (sam) yeah., and I think that for specialist engineers,, which is everything, that's not a generalist software engineer,.

A

One of the most successful models is having them kind of act like internal consultants., either they're owning an infrastructure, service, software and systems, people, or they're, owning a product.

A

and you're kind of embedding for a tour of duty, right?.

A

We'll come help bring this product up to snuff so that it can be supported in production.

B

I was recently in a kind of debate with a few different people about services versus micro services, and does it mean if we all go to services.

B

We have one or two engineers on call constantly, because they own this small service,, the only people that know about it and they're on call...

B

and one interesting part of the discussion. Was--.

B

If you're half way between big services and small services,.

B

That grey area means there's a lot of known knowledge.

B

Just stuck in one team.

B

and one of the arguments--, and I haven't quite decided where I sit on this kind of philosophical debate--.

B

But one of the arguments was: if you boil services down so small and they're very easily understandable and [parseable], in a large organization, you can-- (charity), that's the dream. yeah,. You can just get to the point where you respond to alerts based on heuristics.

B

Around the infrastructure and request failing and you can very quickly get up to speed on the entire service without actually having not worked on it.

A

So I don't think that it's actually so much about how quickly you can get up to speed.. I think it's about how well you can get up to speed.

B

A

One of the things that we were talking about earlier was the idea of explorability, the idea that you can use a tool like scuba or honeycomb, or whatever, to ask questions and interrogate your systems.

A

What I observed at parse and at facebook was that unlike--, okay, so I'm very experienced with the scenario where I come in as the first engineer,, and that means I can pretty much debug it better than anyone forever.

A

I love that position of power. (chuckles), it's a great position of power,, but it sucks for everyone else..

A

It really sucks to join a team and go, "I can never be the best debugger..

A

I'm never even going to be as good as the people who have been here., they're trolls under the bridge, and I can--", and it speaks to the poverty of our tools, honestly, and the mismatch between how we should--.

A

Most people right now, when they're trying to understand their systems and debug them,. They take a thin layer of tooling their dashboards, state of the system, and a deep iceberg of intuition.

A

That they've built up from interacting with that system over the months or years., and you have to have it.. You cannot like go over and join someone else's corner of the company.

A

And know what's happening, right?

A

because the tooling is thin and the intuition is deep.

A

And we have to invert that., we have to-- one of the things that I'm really passionate about is building-- alright,, don't kill me, but I'm going to say it-- social network for debugging.

B

A

But taking out of people's heads, out of their intuition.

A

And making it so that-- okay,. So let's say I'm on call--, I don't know cassandra,, but I get paged about cassandra--, I'm just like...

A

(sighing) "well, fuck. christine got paged about it and she had a problem kind of like this. A few weeks ago.

A

I'm going to go. Look at her .bashrc effectively for visual debugging.".

A

And look at not only what she thought was worth documenting.

A

But like almost an emergent property of how she's, interacting with the system.

A

I know some people who are like, "everything. I know about unic system administration. I learned from reading other people's .bashrcs, .bash histories..

A

We need to make that history way more accessible..

A

We need our tooling to be letting us actually interrogate our systems,.

A

And in an iterative way,, just go, "what about this?.

A

What about this? oh,? What about that?" and teasing it and playing with it.

A

And we need that to not be ephemeral, for it not to just vanish.

A

I feel like we should be able to bring everyone up to the level.

A

Of the best debugger in every local system.

A

and I've seen-- at parse,, once we got everything into scuba,.

A

It was totally possible for an android engineer to come over and debug their own database query.

A

Because they're using the same--, what erects, silos? it's tooling,, right?.

A

It's barriers between systems that are made by,, they're erected.

A

By tools but they're only applicable to one system,, not the entire system.

B

One of the ways we've got around this in our culture is doing everything in chat,, so during an incident--.

A

(Charity) yes. so much!.

B

I'm sure some of the audience will be familiar with this, we've talked about it a few times, but we have hubot and we have our chat rooms, and during an incident, people are not just sharing graphs, they're calling graphs.

B

They're pulling these in, they're pulling in logs, (charity) absolutely. results from like failures.

B

coming into the chat in that context, and everyone was diving in together.

A

(Charity) and your conversation become like searchable history.

B

Right, exactly., so it's everything happening in time.

A

Yes. and it's--.

B

I remember when I first joined github as an engineer,, I started going, and it's like, "oh,. We learn and teach by showing.".

A

So you just ask your question in a chat room.

B

And they show you how to run something., and it's in that chat, room, forever. sure. and you learn from-- (charity). I don't believe in runbooks,.

A

Or documentation,, not really. (chuckles), I don't believe they exist.

A

as a starting point,. Sure. show me the places I should start.

B

But don't try and tell me that you've captured.

A

How I can debug a problem or think about it., it's supposed to be a creative discipline.. If it's not creative,, I shouldn't be doing it..

A

The robot should be doing it., but I do believe that our tools can do a much better job of capturing the rich history of our interactions with them.

A

And yeah, everything that I'm thinking about and building is--.

A

This is not new., it's just taking what I've always done at work,.

A

With irc and graphite, and then it was hipchat.

A

And scuba, and now it's just like, "okay, what, if our tools weren't fighting us in this?"?

A

What if we just tried to do the way good engineering teams actually work.

B

Right. so a subject I really want to get onto because I know you have opinions here.

B

Is serverless., perhaps a silly term-- we're not without servers,.

B

But I think there's something interesting: that the movement, and the way the tooling and the frameworks are shifting towards this kind of model.

B

Is telling us something.? What do you think we have to learn from this approach to provisioning and services?.

A

Yeah, I mean this is my third year speaking at serverlessconf,,.

A

And the first two years I just showed up and threw shit at them,, I'm just like,.

A

"You're wrong, you're stupid.. You have to--", not the most constructive,, but I was just really annoyed by the word.

A

(Chuckles) but what's interesting is they're the ones who were out there on the frontier.

A

Of event-driven systems, and really looking at gluing together, services,.

A

And that glue is the product.

A

and they have no concept of a thing that they can ssh into.

A

So they have to be thinking about...

A

About their instrumentation at the abstract level that we all should be, but we don't because we lean on our crutches.

A

And old ways of doing things., and I think that they're starting to pull together some really interesting, things.

A

Everything's are still fragile.. We used to do our end-to-end health check through lambda, until it wasn't robust enough..

A

It was just paging us all. The time. we're just like, "oh,, maybe not.", but it'll get there.. I believe that.

B

So it's an interesting way: of, yeah, simplifying--.

A

(Charity) oh, wait,, I'm not supposed to talk shit about companies-- sorry.

A

(Chuckles) I was warned. (chuckles).

B

So systems are getting more desperate,, more complex.

A

(Charity) yeah, we're all distributed systems, engineers, now.

A

(sam) right., and this is something where--.

A

Web and mobile were some of the earliest distributed systems. Engineers.

A

The way that you have to think about just accepting failure.

A

Failure should not be rare., it should be common., it should be comfortable., it should be small., it should not be visible. and the way that we do this is by not investing.

A

All of our scarce engineering cycles into making staging perfect all the time.

A

Invest them into testing and production..

A

I'm not saying don't test before production.

A

definitely test before production., but I feel, like that's, been a real place of brittleness and fragility,.

A

Because we become so rigid.

A

we're like, "oh, we're going to make sure it doesn't break." no,, just ship it with a feature. Flag, turn it on for 1% of internal users.

A

And see what happens, you know. there's a set of muscles around that that--.

A

Who here has canaries that automatically promote based on health, checks?

A

one? sweet, two. amazing. well,, it's more than I expected, honestly,, but it's not good enough..

A

More of us need to have those.

B

You've written a book.

A

(Charity) oh, a book, yeah.

B

Yeah, about a subject, that's dear to my heart: databases and reliability, engineering.

B

Yes. a similar question to the one I asked earlier--, what led you to want to collect your thoughts around the--.

A

(Charity) oh, I didn't. honestly,. I get dragged kicking and screaming into everything I ever do.

A

And my co-author, laine,, who is amazing, seriously wrote--.

A

Because I went and started a company right after that., so it's like,, you know, 37 year, whatever.,.

A

But one thing that I did really lobby for that. I'm glad we did.

A

Was name it database, reliability, engineering.,.

A

I've never liked being called an sre..

A

I feel, like that's very diminishing. (chuckles). It sounds like I just keep other people's shit running.

A

But I actually build things.. The one place where that is less and less true is in the databases, side.

A

Database, reliability. Engineering is absolutely the right team. Name.

A

Yeah, I feel like the priesthoods of networks and databases.

A

Have been kind of the last two to join.

A

The shared engineering organizational, in terms of tools and practices-- oops.

A

But it's time,, you know.

A

If you're not sharing tools and practices with the rest of the org,.

A

You're holding them back. they're, holding you back.

A

And reliability and paranoia is absolutely your job.

A

B

I watch a lot of what our database team are doing and they spend the majority of their time running software.

B

To solve the problems that we have, running, really rich tooling around these issues and they think very much as software engineers.

B

and we're going through this journey right now, where it's all soft--.

B

Everything we're doing is software engineering, towards some sort of goal., and you hit this scale and you're no longer necessarily operating.

B

And pulling levers, but you're building automation that solves the problem that you have.

B

We have another open source project that we're announcing this week from the database engineering team.. So there's loads of software coming out in those areas.

A

B

And so to see the book, to see that there's more conversation around database engineering in those ways,.

B

We're moving from administrators,, I think, to engineers, and I think that's an interesting phase that we're going through.

A

I do too., I think that is what an engineer is. an engineer, writes software.

A

None of us are administrators., we can't be..

A

I think that the...

A

perspective I'm coming from always is that of an accidental dba.

A

(Chuckles) I've accidentally always wound up in charge of the databases.

A

I'm not sure why that is., but once you start, it's hard to stop., it's just that fun.

B

Yeah, I had the exact same issue. yeah..

B

I joined a company as an-- (charity). I think what it is.

A

Is it feels real,, it feels important,. It feels vital.

A

Like fucking networks is like-- what,, you drop some packets?.

A

They're, like grains of sand., who cares? make some more packets., so I like that. (chuckles), so I like that with data.

B

You can put companies out of business.. I enjoy that feeling. Of--.

B

Every dba or database engineer I've met fell into it. By--.

A

(Charity) yeah.

B

Gravitating to the weirdest problem that no one else--.

A

We have god complexes is the other way of saying that. yeah, it could be. yeah,. I hold your future in my hands.

B

So we're getting towards the end, but we don't want to--.

B

I would love to get an idea from you.

B

What's exciting about the future of technology?

B

the things you're seeing now,, where you feel they're going to go,, where you think we are going as practitioners and as an industry,.

B

And what's getting you up in the morning to go and look and think about the future?.

A

So I feel like devops is no longer cool,, which is great.

A

That means I can like it now., but it's time for phase two.

A

we're a couple of years into it. phase. One was very much about ops, people., it's like,, "alright,, ops, people, time to learn to write. Code.

A

you've got to write software." and everybody went, "okay, okay,, write, software." and now phase two is like, "okay software engineers, time to step up.".

A

Time to learn to write, operable services;.

A

To understand how to instrument your code; to take control of your destinies at a high level.

A

It's not okay, anymore, to not understand.

A

How to do graceful degradation of service.

A

how to not architect for resiliency.

A

I think that these are really exciting. Problems.

A

systems are increasingly--, they look more. The electrical grid.

A

Than a lamp stack., the problems that we have are bigger., the things that are demanded by our customers are bigger.

A

And more exciting. and I think it's neat.

B

(Sam) so as we move from the first phase to the next...

B

You see that, like in phase one, it was about the ops people.

B

And I think the tooling,, the practices and the attitudes reflected.

B

What ops people wanted to see in the world.

B

(charity) yeah.? How do you think-- so there's going to be a meeting in the middle when we get to phase two.? How do you think that's going to change those tools,, those practices?.

A

Where's that middle ground, and what do you think it will look like?

A

so, first of all,, I observe a lot of software engineers.

A

Meeting this with dismay instead of joy and I get it.

A

Like ops, people did too., but it's better..

A

It is, it honestly is., there's so much power and there's so much...

A

...It's exciting., these are powerful set of tools.

A

and it doesn't have to be miserable,. You know?.

A

I get why nobody wanted to join us being woken up, on call at 2 a.m. everyday.. I don't either!, I'm over 30 now,, I'm done with that.

A

right? no.. We all want this to be better.

A

We want it to be more human-centered..

A

We want to-- the thing that I'm excited about is...

A

It takes all of us but we're building tools that are consumer quality, which is a fancy way of saying that developers are people too., which is something that's maybe controversial, but I actually do believe.

A

and our job shouldn't be miserable, shouldn't be killing us.

A

We should achieve some sort of balance., I'm a workaholic, but it's fine.. I can see this can be happening to other people who have children.

A

I like that., it feels like we're growing up as an industry, and I think that we should all be embracing it.

B

(Sam) yeah,, so we don't want to end up adversarial.

B

if you had a message for either side, so first for the developers,, then the ops people,. What do you think either side are not hearing from the other?.

A

Ops people need to greet this as not a loss of power.

A

but as gaining impact, right?.

A

It means you now have more influence and you don't have to take it on yourselves..

A

You can do awesomer things with your time, right?.

A

And for software engineers,, it's just you know...

A

Embrace the idea that you need to understand how everything works.

A

And have some ownership over it, because it's a really virtuous feedback loop and you want it to be tight. you're, not doing as good of a job as you could.

A

If you weren't getting that feedback loop.

A

And you're going to need event-driven explorable systems.

A

So you should definitely go like buy honeycomb or something.

A

I honestly-- this is coming from a place. Of--.

A

It's a cry for help. (chuckles). I think that the distributed tracing stuff is really super. Exciting.

B

They're, the only other people out there that are really experimenting.

A

With the event-driven space.- and there needs to be a lot more,, because the problems that are coming are really hard..

B

(Sam) awesome. well,. Thank you very much for being here.. Thank you. Everyone for joining us. (charity). Thank you for having me.

A

It was really fun. and thanks for matching my hair, (applause) ♪ (jazz music) ♪.