Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF SIG Observability Updates - Bartlomiej Płotka, Richard Hartmann & Simone Ferlin

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

CNCF SIG Observability Updates - Bartlomiej Płotka, Red Hat, Richard Hartmann, Grafana Labs & Simone Ferlin, Ericsson

The CNCF SIG Observability started in Summer 2020 to gather contributors and users from different areas passionate about Observability topics. During this talk, we will be giving an overview of work done, work to come, and areas where you can get involved and help. Consisting of open source project maintainers, standards authors, end-users, and more, we cover a wide range of experience and invite you to join us as well! Help us make the CNCF Observability space better, diverse, and more approachable!

A

Hi and welcome to.

B

A

Observability updates. We have three people today.

B

Simoni freli I'm an observability software architect at erickson in sweden and a senior at the junction lecture at the university of cal state in sweden and a sig observability member.

A

Hello, richard permitted team member openmetrics, founder sick observability teacher and I work at grafana labs.

C

Hello, my name is bartik plotka and I'm a principal software engineer at red hat. I am a prominent use, maintainer and co-author of thanos product project, I'm sick, observability technique and I also write book about programming with aurelie. So let's get.

A

Started what is all of this about? Well, aesthetic is a special interest group. Most of you probably know this and observability is fundamentally about making complex system understandable for humans. There's.

B

Other definitions, but.

A

Like the distilled version of this is usually this that, as systems become ever more complex, you need to actually be able to understand them as a human and that's what we're working on what this also means, as we are sick- and this is a group that we want to talk to you. This is also true for this talk, which is why we'll try and optimize towards keeping it relatively short, to have as much q a and open discussion at the end of the slot look under our charter.

A

We are basically there to shepherd the observability space within within cncf and help different projects different end users. Different vendors come into the space and interoperate nicely.

C

Okay, so let's quickly recap what we did from really the beginning of the cncf seek observability creation, which was a summer last year, so the most important duties that we uh we were told to do is the review of the projects that are being proposed to change the stage um of the cncf. um um Where.

B

They are in the.

C

In the in the cncf landscape, so stand box, incubated or graduated stages. um So what we did? We submitted four uh reviews for four projects, uh two of them cortex and thanos, and ended up being incubated and uh we kind of spent a lot of time to review, open matches and open telemetry uh which are being passed uh and the review has been passed to the toc and are awaiting their decision, um so that was kind of a large chunk of our work. But, let's now look on you know our.

C

You know overall high level landscape of the observability in the cncf for better wars. We still are you.

B

C

This old-school model of observability free pillars and that we we used to know which is metrics tracing and logging, and the reason is that everything costs right so, um for example, metrics, while adding less context that you might want to to to have um in terms of uh you know unknown things that you you should observe, um still are worth to have because of the just just the lowest cost for um collecting those and visualizating, um and usually that's a start for your journey.

C

In terms of you know, incident recovery and and finding out what was the risk of that incident and what is the overall health of your system right? So this is what you use for, alerting and for immediate triggers um or calculating slo reliably over time. Right and usually, this is a starting point for the incident duties and with the help of the logging.

C

This is how you start your debugging investigations and, typically you would end up kind of going into tracing only if needed, to gather more crucial informations about the you know, their life cycle of the requests that happen, sometimes between microserve, multiple micro services in your cloud, um and um you know that used to be our landscape, but it is right now uh still the case, and uh this is what what people use on um on production, but that's not at the end. Right like there are things beyond that, and- and we are, it was predicted.

C

You know already two years ago uh that you know there will be more signals at the end. You know something that clearly uh has appeared is something like continuous profiling, where you want to know your performance over time, and you want to know uh you know some historical performance. You can compare things you can uh compare in.

C

You know kind of time spent um cpu time spent per every function or memory allocations, and this proven to be very, uh very valuable and there are in the open source communities project that that allows you to do so, for example, control and at the end there might be more like crash dams. This is something that um I think is very valuable for for richie and, and I think it would be available for everyone but um yeah. This is some.

C

You know uh million dollars, idea that that we are still waiting to to to have someone to implement it and within the cnc, app observability. We also have some methods to activate more information from your system. You would say that those are kind of the testing practices that are maybe a bit novel in the cloud native. um You know foundation, and this is essentially you know, advanced testing like fuzzing, or maybe something that I would say it's uh fuzzing on production right, which is essentially cows.

C

Monkey techniques, cal, state uh cows, testing, where you are randomly uh triggering uh fail uh over scenarios uh in the random parts of your system to ensure stability and- and you know, good incident reactions.

C

So now, when we look on the you know, project landscape we've seen some movement from the from the last year or from there, even from the last cube in uh north.

B

C

So we well first of all, we we noticed there are.

A

C

Projects, cortex and thanos, and that joined open tracing, nothing changed on graduation site, fluency yagar promotions are still very prominent and and still getting more adopted, and on the sandbox side we see a new project trickster, which is you know, known from amazing caching capabilities on top of promotions.

C

We have open metrics and open telemetry which were being proposed to get incubated and we still have cow smash and litmus which are our. You know, chaos, testing projects we have here.

B

So, what's in our to-do list in the sig observability, so we have some activities in progress, uh mainly a white paper about observability.

B

So all all these stories and telling a little bit um talking about the foundations of observability, like like bartek shared with you previously, including other topics and, most importantly, we are including everybody in the siege and inviting everybody to to contribute to this document, because we see that there are people coming from different areas from different industries and they might have a different references and also different input to contribute to this document.

B

So our goal here is basically have a cncf stamp or, let's say a base document when somebody comes to the cncf and wants to learn more about observability what it is and how do I get from zero to hero and what what exists in the landscape?

B

This is at least one outcome we want to have in the sig. The second document is best current practices on tracing for from the end user perspective. So if you remember the triplers that bartek talked about, um so we have logs metrics and we have distributed tracing so in tracing.

B

You usually need a little bit of education on the application development side, so you usually have to learn how to instrument your code and which information is relevant from your code to be distributed with, what's relevant, to be propagated in a in a distributed context when you have several micro services talking to each other.

B

So this is a little bit what we want to share in this document, we have a smaller set of people from the sig contributing here, um usually people that already went on this path and learned from mistakes and learned best practices, and they want to document and share that so and as a backlog, we have some activities mainly webinars on observability.

B

And a white paper on overlap of observability and a I n ml. So there is an l missing in our slide, but we we do want the machine and learning aspect of that so um and a little bit as input so learning ways of working.

B

We are trying to understand or trying to to have a little bit of a way of of working that comes from, let's say well understood or well operating communities such as the ietf, if somebody is familiar with the internet engineering task force.

B

So how the the process of coming from an idea from a project from from code to having a standard that you can share with um other vendors, other companies and other developers, and they all need to interoperate so in in the cncf or at least in the sig. We are trying to think about like having a clear distinction like, for example, in the irtf and iatf.

B

What is experimental and more research oriented versus what is more engineering driven problems and implementations?

B

You can have implementations in the experimental and research oriented part as well, but trying to distinguish from from where the the let's say the impose is coming in the community is something it. It's an engineering driving problem, that's happening in a certain industry in a certain part that maybe it's not the community that I'm coming from the industry that I'm working for um also have a little bit more of a process such as the the request for comments in the ietf.

B

So, like a document that, basically, if you want to implement a protocol, you have a document that describes or not one but a set of them and these documents there is a it's not that simple. So there is a process, and sometimes a very long process to go from a proposal to have a standard at the end that somebody can follow and can rely on that standard to be interoperable with another system.

B

So how? How does the yeah? How should we better implement that? How should we better work? um Should we copy this this way of working? Should we implement something different? Is it interesting to have proposals first in a document? Have people who propose like having different implementations and trying to make their code to interoperate and then based on this interoperation?

B

You refine the documents up until the stage where you have a solid, more solid document, and then you could have an ifc.

B

So these are some things, um we're thinking about, and one thing that um and of course, I'm not not only trying to implement or copy what's working, very well from other communities, but one thing that could work very well in the same way that works in the itf is also the mediation between projects and vendors. In a sense of the goal is to have a standard.

B

The goal is to have interoperable standards and regardless of from which project you're coming from or from which vendor you're coming from the goal here is to have code, that's running and code that interoperates.

B

And is, let's say: vendor neutral, so create a community. That's based on the technical problem that you need to solve. You need to implement and it's less dependent on interests that could be coming from a community from an industry or from a particular vendor.

B

Well, there are just three of us here, but all the work being carried out in the sig uh observability is there are lots of helping hands behind us, so the list of people are here and apologies. If we forgot about someone, but we really try to include everyone here that participate in the calls and are active in in the sig.

B

So thank you very much and if you want to participate, if you got interested um in what we shared here with you, so we have our fortnightly calls. um The link here is here in the in the slide. We have our repository. Everything that we talked about and we we share in the siege- is public.

B

We have a select channel as well, where people ask coming ask things and whatever a mailing list and if you have free time to spare and would like to share with us or no so thanks, everyone uh come to the seek observability and yeah. That's all from us for now.

C