GitHub GitHub Universe 2020: Developer - DevOps, 10 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Observability: superpowers for developers - GitHub Universe 2020

Description

Presented by Christine Yen, CEO/Cofounder, Honeycomb

The adoption of observability practices has allowed us to understand our production software in a whole new way. When viewed from the perspective of a developer, it can feel like gaining superpowers. When observability is folded into the development process itself, it represents the potential for a beautifully virtuous cycle: Production stops being just where our development code runs into issues, and it becomes where part of our development process lives. This talk will cover how to implement observability practices (from your tools and from your humans) and what you can achieve as a result.

For more from GitHub Universe 2020, visit https://githubuniverse.com

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

A

Every time I think, I'm full up on the superhero genre, a new movie comes out and sucks me right back in. uh I think, there's something irresistible and appealing about this fight against chaos, about good people trying their best and having to level up fast and segway. It could also be pretty reliable relatable to our lives as developers pushing our code into the chaos of production, relatable, maybe minus the hollywood actors and glitzy production, let's jump in so, as we all know, all good superhero tales start with an origin story.

A

Here's mine, I'm christine, and I am a developer early in my career. One of the things I took pride in was that I was fast. Sometimes it seemed like I was non-stop. I could just go.

A

Go go, get this little cycle going and I made it through a few jobs actually before I interacted with with my first real ops person, and I met her because I was working on a new component of a system and that nice little development cycle, I'd gotten used to started to look a little bit more like this, and it became clear to me quickly why certain unflattering stereotypes or around engineering roles exist, which is such a shame. We could be on the same side.

A

Theoretically, we are on the same side we're just trying to provide a great experience for our users, but what we were doing hunkering down, refusing to see the other side's perspective, blaming them for some failure or downtime just made it harder to get that done since then. Fortunately, the world has changed in a way. That means whether we like it or not. Dev and ops have to learn how to work more closely together.

A

The days of having one database and one monolithic application running on one server or a handful are gone. We've decoupled and decomposed everything to the point where no one person or an ops team in the absence of developers can make sense of it anymore, and this old pattern of developers write code on their machines. Ops worries about it after release, it's already started falling apart and in its place, teams are building a shared sense of ownership of production.

A

How well, if the first wave of devops was for ops folks to learn how to code or to automate their work.

A

The second wave of devops is for developers for us to learn how to own our code in production and that little bridge, my favorite way to blur the lines between dev and ops is through observability, now, there's an old school definition of it, which is useful when talking about disciplines like spaceflight mechanical engineering in today's software world, I like to think of it, as why is my software behaving like that over there observability is more than just a tool chain, it's about the culture and processes of a team and how the entire engineering org thinks about production.

A

So why is this especially interesting for developers? Isn't observability like monitoring and more of an ops thing, I say no observability is relevant to developers because in a way we're already doing it, we have the most to gain.

A

So many of the right instincts for catching problems in production are already in place. If we look at our modern software development process, it's already so full of curiosity. Just look at the number of times.

A

We, you know include this concept of testing, comparing the code that we think that we wrote versus how it actually ran, and we do it again and again, as the environment gets more complicated.

A

Even this last step, something that previously fell on the upside of the deployment wall can be considered a form of testing writing pre-production tests. Good pre-production tests is all about thinking through various scenarios right describing how our code behaves and then investigating when it deviates from those expectations.

A

The more practice we get at it, the better our code gets because we begin thinking ahead and explicitly examining our assumptions.

A

When folks look at graphs of how systems behave in production, they're, basically doing the exact same thing, trying to figure out why the nicely behaved system that we expect is behaving a little bit more like a whole different beast and as we transfer what we do in our written tests to graphs describing what we're doing in production, we will be the ones who will be best positioned to handle it when expected, deviates from actual and then suddenly what ops does in production?

A

What ops used to do versus what devs should do doesn't feel like such a sharp line. After all,.

A

Okay, so we're bought in we've set the scene. We know our players now we can activate our powers earlier when I described observability you'll notice. I use lots of verbs understanding asking questions, because not just having a tool isn't enough to let you do the powerful thing, there's a few things that we need to do or can do ahead of time to make this easier for the developers.

A

First, one of the first things to really empower developers in production is to teach production tools to speak our language, traditional monitoring tools, tend to assume their users care about a certain subset of nouns and they often don't tie directly back to the concepts I deal with in my code.

A

Having an ops person come to me angry that cpu utilization is up on a particular subset of cassandra nodes tells me nothing about how to go about diagnosing the issue, much less, how to fix it. In my code.

A

Compare that to the sorts of nouns that appear, naturally, in my tests, things that I know will trigger different behaviors in the code itself, if, instead of getting alerted about cassandra right through it, what if I found out that the latest deploys seems to result in increased latency for a payment endpoint and our second largest customer was impacted.

A

Well, that gives me a much better starting point for reproducing that behavior, possibly in a test and fixing that issue high cardinality data is also key.

A

Those fields that hold a ton of potential values again, these are the names I deal with day to day that capture characteristics of the workloads that I'm using that change, how my code behaves- and these are the nouns that I'll want to slice and dice by to understand who's being impacted and how to go about putting together a fix to ground this in reality, build id it's one of my favorite stealthy, high cardinality attributes, right theoretically, unbounded set of values, but also one of the most immediately useful.

A

This is a screenshot of our dog food instance of honeycomb, and it is a data set that represents our traffic hitting our web app you'll see the second starter query that I've circled that our engineers have configured is just one that asks has the code I care about been deployed yet super simple and often hard to do hard to get in our traditional monitoring tools.

A

Finally, we should treat instrumentation like docs and tests. We know that the easier it is to write and run pre-production tests. The more folks will write them the easier they'll be to update. The same is true of instrumentation.

A

All of these things lead to developers being more comfortable with production and production, not feeling like some far away magical mysterious place anymore, but instead just an extension of our development environment.

A

So we just talked about a bunch of nouns making sure those line up, but we can reach for different visuals too, to help map production artifacts back to developer world concepts tracing is one of the most developer friendly evolutions. Yet in this industry you look at a trace.

A

You can immediately start to map that to your mental model of how code executes back to developer world concepts, someone recently described tracing as um chrome, developer tools for non-browser software, which I loved, because it's true and it's so exciting- to be able to connect this very developer, friendly final representation of something happening in production, where I can see surface boundaries and function, names to the graphs and higher level analyses that let us understand how our code is running in context.

A

We need graphs to be able to pick out patterns and understand what happens when we run that code over and over and over again, not just in a test environment and having both being able to switch back and forth between the two is. What allows us to take that production sense and map it to what feels natural as a developer.

A

And tracing has cleverly found a way to make logs tractable and meaningful we're not in a log scale, world anymore. All right logs are no longer human scale, anymore, everything's machine scale. Everything is huge once you add a little bit of structure into your logs, some semblance of hierarchy to tie individual log lines together, and you start to get something that again better reflects the code feeds our understanding of production tracing is what happens when logs grow up.

A

So what does this all look like? How does that virtuous cycle make me better at doing what I actually care about writing code? Well, let's run through, let's run through a few examples.

A

When observability is our superpower, we can use it to figure out what code to write in the first place right when we start our days it often or when we see a ticket or get started with a new task, often there's a frustratingly vague complaint. Something is slow, something is broken.

A

Okay, what's slow is everything slow are only some operations slow, sometimes for some users exploring and investigating these sorts of questions in order to figure out where to begin with, our code should feel like the way that we investigate anything in life by asking new questions and exploring starting at a high level, identifying differences between expectations and reality and then teasing apart that signal from noise, iterating and zooming until we can isolate something interesting and then being able to flip into that specific example to understand how all the building blocks fit together.

A

And if we do that, we can stop guessing at what we think will fix the problem we can. We can start to map data to okay. This. This seems like the right place to focus our efforts.

A

These are some tweets from cj silvera over at ease, who recently got our team to invest in instrumenting a legacy monolith and suddenly realize what it was like to be able to see the more you can understand. What's actually happening with your code, the less scary it is to think about changing anything about it and by embracing observability we can. We can learn what to do with our time for maximum impact. It's no longer a scary problem to touch this thing that other folks built.

A

What observability is our superpower? We can use it to figure out the best way to write that code again going back to the sometimes you get a ticket that just says something like build out. Nessa json support, okay, cool. Well as a developer. We know we can just build it, pull it out, but first how many users are actually using this.

A

How might it impact them? Is this what we want who's relying on the old behavior, by by being able to ask these questions to understand what normal is before? We change that normal with our code, we can figure out if the fix will actually be a fix.

A

All of these domain specific pieces of metadata, I talked about earlier they're. What help us understand? Does it matter to the business? Does this matter to my users?

A

Instrumentation basically turns production into one giant, machine, spitting out debug statements, but just against real world traffic.

A

You know some um some folks out there might be sitting and thinking well. This is this is what my product manager does right. We're product managers out there thinking about what to do and- and uh you know, building their plans based on intuition and qualitative interviews.

A

Well, we're developers, we have access to prod. We can make arguments based on data.

A

When observability is our superpower, we can use it to even anticipate issues and predict the future complex systems. Our systems today have an almost have an infinitely long list of almost impossible failure scenarios.

A

Our systems are only getting more complex, which means that our previous approach of trying to guess at the future, enshrining guesses into pre-production, test cases or dashboards on a wall metaphorical wall in this world isn't going to be enough.

A

We have to move away from trying to predict each of these incredibly rare, extremely unpredictable black swan failure scenarios and embrace observability to make sure that if something happens, we find out immediately and have enough information to know what to do about it.

A

Test and production has become a polarizing phrase for some, so necessary disclaimer. This does not mean to not test before production, but production is a great test bed for hypotheses.

A

I'm going to tell a story about our friends of right, gecko board, who we're building out a new feature that essentially reduced to the bin. Packing problem right, an mp complete problem, lots of possible ways to solve it. um No one, you know necessarily blessed best solution and instead of having an engineer or multiple engineers, kind of go off and build something and test something and run simulations and try to guess at what would work best for their use case.

A

They came up with a few different hypotheses, three different algorithms that they put in product. They ran production traffic against, captured the output stored. The output threw away the results, no user impact, but they were able to measure how each of their experiments ran. They were able to, by the end of a day, say hey. This one looks like it's best against our ship of traffic and our current customers. Let's move forward with that, and all of this is possible because their tools supported these ad hoc experiments.

A

Their team believed in flexible instrumentation and their development team had gotten comfortable running experiments in production.

A

Finally, I love the superpower because it pairs so well with these other techniques that are becoming commonplace in our modern development processes. Feature flags are great: it's incredibly powerful to be able to test out code on a very small amount of traffic, it's even more powerful when we can take those feature flag, values on off. You know these 15 feature flags all converging, take those values and pump it into our tools.

A

We pump it into our dog food instance of honeycomb, and then we have access to all of our normal top level metrics, but are able to slice by whatever tiny segment of folks have been flagged into this experimental feature or that experimental branch that day.

A

So this being a talk and not actually a movie, we are jumping to the end here. um I'm sure you've all seen your fair share of cgi battles where good times over evil.

A

In the old world, devs cared about code in the production, sorry in the development and test environments and ops cared about prod, but there are all sorts of reasons now to break down these barriers and empower developers to own our code and production.

A

We talked about some concrete things to do to facilitate that today, so that we can all take our learnings from production and supercharge our development process at each step along the way. There are things that we can be learning about how our code is behaving in production that will inform our choices and ensure better output.

A

And, as we all know, from the marvel universe, being a superhero is more fun, with a team in real life being able to transfer that knowledge and share ownership, not just a plot device, it's how we scale good intentions and build intuition.

A

After all, all debugging is inherently social. If, even if it's just present you trying to figure out what past you discovered in order to help future you and that's why I'll hammer this point again, observability isn't just about your choice of tool, although that can catalyze change.

A

Ultimately, it's about the processes you enable and the culture you build around production in the end, we're all just trying to fight fewer battles, we're trying to reduce that tension and release to reduce the number of times we're woken up in the middle of the night or our teammates are woken up in the middle of the night, because that's how we burn out that way lies alert, fatigue too many dashboards and the sorts of habits where new engineers have to decode and claw through a bunch of tribal knowledge to learn which notifications to take seriously, which ones can be ignored.

A

The more work that we can do to shift high stress situations where we have to be reactive and then stressed into proactive work where we can think ahead of time about expecteds versus actuals, where we can build up those muscles to investigate issues. The more we'll be able to take care of our teams and focus on doing what we want.

A

So if you happen to be an ops or infra engineer out there think about how you can empower your devs to share the great responsibility of production and the great power and, if you're, a developer, remember that these superpowers are possible and have been within your grasp this whole time we stand to benefit the most from observability, because it'll help us write better code at each step along the way you um I'd love to answer, questions or point folks to more resources for anyone looking to learn about more about observability.