Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Prometheus, Jaeger, and Envoy: The Observability for Today's Enterprises - Priyanka Sharma

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Keynote: Prometheus, Jaeger, and Envoy: The Observability for Today's Enterprises - Priyanka Sharma, Director of Cloud-Native Alliances, GitLab

What do Nike, DigitalOcean, Grafana and Red Hat have in common? They’ve all adopted multiple major open source observability projects over the last few years for DevOps. Priyanka Sharma explores the various projects leading the way (including Prometheus, Jaeger, and Envoy), explains how they fit together and offers a view of the future ecosystem. In addition to discussing the current movement, Priyanka shares best practices on how organizations build on and contribute to observability in open source, which will continue to strengthen the ecosystem.

For more info click here: https://sched.co/FzDg

A

Thank you for that introduction, Janet Nihao Shanghai. How is everyone this is such a wonderful city, it's my first time in China and I'm so thrilled to be here today we are going to talk about observability for enterprises, Prometheus, Yeager, hto and other projects in CN. Cf can play a big role in your observability strategy and we will be covering that as well.

A

So, as Janet said, I am priyanka sharma and I'm. Director of cloud native alliances at gitlab I also contribute to the open tracing project and have recently started working a lot more with the Jaeger project.

A

So the thesis for today's talk- the main point I want to get across today, is that the open source ecosystem is teeming with projects related to observability and in this landscape, where we're all living and learning and it's building together, it is imperative to understand the top projects and build a framework for utilizing them together with the cohesive strategy for observability that focuses on meeting your business goals before we get into the meat of the con.

A

The conversation I'd like to thank some people, so Cindy sridhar and she I think of her as an observability badass she's written some great content on various nuances and perspectives on observability and I personally have gotten great value out of her work. I reference it through the talk I'd, also like to thank Richard Hartman, who works on prometheus and open metrics Richard is here in the conference and he's going to be talking about Prometheus, I, think tomorrow, or something like that. So we've had conversations about this topic and I've drawn from from his insight.

A

Finally, Ted Jung is the lead maintainer for open tracing, Tim and I have worked together for a while and Ted thinks very deeply about the complexity of distributed systems at scale systems. We all deal with and I have incorporated his thoughts into this presentation as well.

A

So what is observability people say that observability is the new fancy name for monitoring to make developers like it more there's a little bit of truth to that. However, in today's modern world, where we are building more complex, more mission, critical software every day, especially with kubernetes as the backbone observe observability, goes a lot further. It is, it consists of three main pillars of metrics, with strong codified alerts, tracing and logging.

A

Why does observability matter actually good good gut check here with the audience? Do people care about observability? Is it important raise your hand all right? Good people are listening, I came from San Francisco for a reason, this is great Thank, You audience so observability matters, because speeding up cycle time is critical, as, as was said before, reacting to the market is important to today's businesses. Kubernetes enables that and an observability staff will make sure that you ship fast and reliably.

A

Business survival, for that reason, depends on a faster DevOps lifecycle, but the problem is we're not at peak performance today, there's been a lot of investment in DevOps and observability, but we're not seeing the results expected. Just in the last year alone, 3.9 billion dollars were spent on DevOps software, but eighty-seven percent organizations are disappointed with the results of DevOps, and this is all based on research done by the likes of IDG Gartner, who are for all the things they do. Research is top notch, so why is that happening? This doesn't mean we're at zero.

A

This doesn't mean the advent of DevOps has meant nothing. We've come a far away from when we just started building micro services and had no idea what was happening in our systems, but the problem is that today's tool chain limits the fast DevOps lifecycle. We really need to achieve the next level. If you think about it.

A

The integration complexity of tool chains slows down teams, so this over here is the various stages of the software development operations and security lifecycle, and you see so many logos and by the way, these are not even close to all the total number of companies projects in this ecosystem. This slide is really stressful and ugly. It's meant to be I mean how many folks here use four or more tools for these various stages. Together, come on people see I'm, seeing a lot of hands. I feel your pain. This is not how it should be. Why?

A

Because the tool chain crisis is costing us a lot. Different teams use different tools and different integrations and here's the price we pay. We spend time and cost to acquire all these tools. We spend time and cost to integrate all these tools and then there's the most insidious cost of all, which is the user context. Switching now, as an efficient, DevOps engineer, you might be, you know, I get an alert on slack, I click it and go to a graph on a dashboard.

A

Just takes me a couple of seconds, that's true and on a micro-level, not bad great work. Devops engineer at github I work a lot with enterprise companies and they have so many huge amounts of developers in their companies. So let's say someone has 6,000 developers for one context which that costs to sack. Suddenly you spent three hours for a context, switch. How many context switches do you have in a day exactly.

A

Oops, sorry and the problem is this is getting worse, because the shift to micro-services creates an explosion of the number of projects that are spun off and the tool chain everybody is using, but a well-thought-out observability strategy that depends on a hub across the organization can contain this complexity and help us ship fast and reliably.

A

Let's go back to the pillars of observability, we talked about so, as I said, there's metrics with alerts, tracing and logging. Why do we have these three pillars? It's not because this is the best case scenario. The best case scenario is an action replay of your entire system, with all of the data going through it in real-time available without any latency.

A

Who doesn't want that here right, exactly the reason we don't have, it is because it's impossible to achieve that in a cost-effective manner, so we do the best we can and we have metrics where we measure what is important to us and get alerts when our service level objectives are not met. We log things for compliance and send all our raw information here, and then we trace transactions that go through the most important aspects of our business, so we know exactly what's happening for these.

A

For these specific cases, this is where we get a little bit of an action replay.

A

We have the the metrics tracing logging trifecta, but the holy grail of system analysis is in a hub life experience where your alerts lead to example, traces. So you can deep dive on the problem and then you can go to the relevant logs. If do you really figure out? What is wrong now? You might notice I didn't start with logs. This is something I'll continue bringing up through this conversation.

A

I know it's our instinct jump into the logs and start swimming, but the most important piece is the metrics and the right alerts that send you to the traces to deep dive, and then you go to logs as necessary.

A

There's many tools in the ecosystem, as we discussed in the beginning. Some mention here like get lab and if you are the hub, like creation, that you need to make it all come together: Prometheus flu and D graph, Anna, Jaeger, open tracing and others. You've heard a lot about them from Liz already and you'll keep hearing about them through the conference, but these can be used in various ways through your observability staff, based on what makes sense for your ecosystem, as I mentioned, you've already heard about these projects.

A

I'm gonna skip through these slides, which I had to describe the projects in brief detail. I'm guessing everyone here already knows what prometheus is gut-check tell me: everyone aware of Prometheus, raise your hand, great work, good audience listening so, as I said great or metric solution, open tracing is a vendor-neutral API for tracing and get lab is a single application for the entire DevOps lifecycle and SEO is service, mesh plus plus, where it connects, monitors and secures your services. It has envoy baked-in, which is what brings all the observability goodness.

A

So now, let's discuss what we started out to talk about in the first place, what are the best practices for engineering leaders as they'd go about building their observability staff?

A

The essential question is not what projects you're going to use, but rather what is your company-wide workflow?

A

You need something that is flexible, so folks can deep dive an instrument as they need, but it's also visible across the entire organization. So people know where a project is in a lifecycle efficient. So you can collaborate without waiting on someone else for an out artificial pass code and govern so DevOps engineers can work without concern about permissions and security, and all that when we think of the traditional versus ideal DevOps and observability toolchain there's the word experience no offense at all to Microsoft here, but the Microsoft word experience.

A

Is there one person edits at a time? There's multiple copies version conflicts. You wait for feedback, it's kind of sequential it works, but it's much nicer if many people can edit at the same time as I said, this is not about the company it was. This is about the workflow you guys, so many people can edit with Google Docs at the same time, there's one copy version control, so no conflicts, real-time feedback and, ultimately, a concurrent experience. This is what we need.

A

So how do you build such a observability stackin culture instead of telling you various options of put this project with that project together, I thought. Let me teach you how to fish and develop a framework that anybody can utilize to take into account their company's specific needs and build out the right, observability staff for them. I will always say that you need a hub like model that goes across your organization. Beyond that start with the external objective in mind, who is your customer and what have you promised them?

A

For example, let's say you're an e-commerce company, then you promised an exceptional shopping experience to your shoppers, irrespective of whether they're used to online or offline but Europe, offering it online. So now you know who's your customer and what you promised them next is to waterfall that external objective internally now I know. There's waterfall is a loaded word in this community, but in this one instance it's actually a good idea. I recommend looking at the top-level objective and breaking it down for each organization, each department, each team, each engineer each employee think about it.

A

So, for example, in the e-commerce example, the merchandising folks might want to provide the best curated options for people to buy. They I want to make sure that they have. The buyers have optionality. That is more than what is available in store on the DevOps side. It might be the best snappiest shopping cart by experience. So when you click the Buy button, it's done once you know that. Then you need to define what data you need on the dev website.

A

Let's focus on that Buy button in the shopping cart, then, if you want that to be seamless, you need to make sure you've mapped all of the transaction as it starts with the click of the buy button goes through with your very fragmented, complex system. The response that comes out you can use distributed tracing for that once you do that, then you'll be able to know whenever there's a problem and ver. It is exactly. You of course need to set up the right monitoring, with codified alerts on the SLO that you care about.

A

You need to know what's p99, what's p95, what's your error budget, because without that you don't have a goal, so you don't know how to give your customer the best experience you promised them once you know that! That's what you need, then you instrument for depth not breadth in the critical areas, the critical area in this case being that click of the Buy button. In the shopping cart, you need to know exactly what happens throughout the stack in that process.

A

Not just a little bit not you know it doesn't matter if you know what happens in the product category pages or even in the home page. If your number one objective is the snappiness of the Buy button.

A

So that's a good framework that you can use when you think through how to build an observability stack now, as you think, through it, there are some opinions or best practices. I should say that I've collected over the years in my with my time and observability, this also adds in the information that I've seen other observability leaders share, which may be useful to you, as you think, through what you do so number. One I brought this up before, but I'll say it again.

A

Logs should not be your first stop. So look I've done this. We have this natural reaction to send all the info to logs and also to just go start swimming the minute. There is a problem, but you need to focus on metrics. First, you have to know what is important, monitor it and then set the lighter right alerts.

A

When you would have a problem, a trace can tell you exactly where it is, and then you should go into the logs to identify the exact issue or if there some things are really confusing and then there's the compliance reason so really try it is. It is unlearning a habit but try to not make logs your first stop don't overburden humans. Now.

A

How many folks here suffer from alert fatigue where you get so many alerts and it's so hard to know which one is important until you've been analyzed it and looked at it in detail, I'm seeing some hands I need more honesty in this room.

A

How many people have alert fatigue all right more folks, guy now I know you're listening, I flew from San Francisco for a reason, so it's important to codify your alert States and only set those up that are in direct alignment with your SLO objectives, rather than letting people at whim add in whatever makes sense this way, you make sure that whenever an alert does go out, it's an it's a turd, a 3:00 a.m. it is worth it for that engineer, to get up and troubleshoot the issue and not just some false alarm.

A

Finally, a service mess. Sorry, a search Wow a service match isn't just for observability, now, I, just touted service meshes as a great way to be a hub. So what am I saying here? Well, you can't just start using a CO and call it a day. It's what I'm trying to say here. You need to make sure if you did a service match is a great way to start with the hub process, so it's great to create a platform for all the tooling that your services will use out of the gate.

A

That's awesome, but you need to know one that a it covers, breaths, not depths. So two, let's go back to that example of the snappy shopping. Cart. In that case, you need to be able to instrument deeper than the service mesh to make sure that that specific transaction is fully understood. The mesh can't do that for you. And thirdly, if you're using a service mesh with four observability, you need to observe said service mesh itself, alright, so to wrap up.

A

I would like to remind you that always think through a hub like model, then building your observability and DevOps stack in general. You need something, that's flexible, so your developers can instrument deeply wherever is important for them and, at the same time, from a workflow perspective, things need to be visible, which means anyone can understand where a project is in its lifecycle efficient. So anyone can collaborate when it makes sense for them, as opposed to waiting for artificial pass, pass. Overs and third is govern, so no one has to stress about.

A

Do they have the permissions? Is this? The right thing is: does the wrong thing they just need. It just needs to work, we're all working towards it as projects and vendors and we're getting there, but with some creativity and think deep thinking you can build this observability stack today. It just takes some time and effort, because 20 minutes are totally not enough about the topic of observability I put some resources together. This is just a small list. There's a lot more information out there.

A

Please take a look and see what makes more sense for you in the observability world and then thank you. I love, observability and geeking out on it. So please reach out and talk about it whenever you like, I'm, always accessible I, would love to chat with you help you, where I can and with that. Thank you so much to Shanghai to this wonderful audience for listening to me, Janet for such a lovely conference and CN CF NLF for bringing this ecosystem together, sheesh yet and enjoy the rest of the contract.