Cloud Native Computing Foundation Observability Special Interest Group, 9 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF SIG Observability 2020-06-09

Description

CNCF SIG Observability 2020-06-09

A

A

A

A

A

Hey dog, she can tell me that you wouldn't be able to make it, but but inside we chatted.

B

A

Yeah, so so we wish a Tiffany a nice time, for example, with what I have been sending you three.

B

C

Unfortunately, between two people, video Mormons.

C

C

D

C

C

C

So let's get started, I guess.

C

So, as per usual, this code is already recording, but everyone is free to join without make without talking or even under a different name. We don't judge just FYI, so you're aware that that all of this is being recorded so.

C

Let's start with Santos cue diligence, I hope everyone read this document who read it picture offense I, see all of you in the period.

E

Two or three people in off the grid on purpose for that even a laptop for a couple of three weeks, almost and I'm just back this week, so I have not read it in others, I.

C

Think we have a federation and last time, I think we also have to two attendees left. So this improves the ratio more, but I think we actually have a motion somewhat decent.

C

So I also throw it into the chat, so everyone can just jump into the document, as with the last of you and I would highly suggest we try and walk through at a certain pace so to make it explicit everyone.

C

Everyone should feel more than welcome to raise objections or anything at any time, no matter. If this is your first call or your tenth call on this call, you should feel comfortable actually putting your your comment or your name or your images. That being said, I'll try and make it quick, so yeah should we can screen share if you want that actually makes sense.

C

Okay, I hope you can see my screen.

C

D

You can see perfect.

C

C

Let's walk through it as per last time. We know that Thanos is self-governing. We know that there is a provision in place to limit takeover exams by a single company. We know that silence has a code of conduct should I open it, or is everyone happy with just acknowledging that this is the case? What I just talked about good, so someone put in the comment section for me. Thank you.

C

So, let's play call for consensus. Big.

C

Sur phone commotion in the background, the little ones are happy, so I call for consensus. Sic observability is happy with the section above all agreed. Anyone not agreed I.

F

Think first one.

F

C

Put this in there: okay, sorry, okay, yep! So for the same call for consensus, my suggestion is take observability is happy with the section about.

C

Very good: are there any production deployments of fairness again, I think most people on this call will be aware of several of the production deployments, but if you're not improbably is still using it yep they're using it in production, and they have quite a large installment or installation I noticed by myself.

C

Also not not small, so long story short call for consensus, same suggestion as before, SiC observability is happy with the section above all agreed. Anyone disagree.

C

C

The next section is about the scenes of principles and alignments with the principles of the scene, ZF, transparency and guidance and technical government's quality. We already talked about the governance above or at least briefly and I. Read it and I agree with that. It's it takes all oopss.

C

That is a duplicate question which we need to attach to TOC to to change the template same here. That's also dupe and same here, that's also good, so we will be walking through those in a second the question. If it's useful for cloud native deployments, I think we can agree with this. Of course, it's obviously following the parameters and that's the kubernetes way of doing things and if affinity for our CTF wants to operate. I would also agree with this, as it's literally the police's way of operating just with objects to an outro screen.

C

So same call for consensus ever once again suggest that we put in the observability is happy with the section above all agreed any disagreements.

C

C

This is about the design of of tennis and that our components will.

C

Denote reliability concerns, or should we just like so now we come to a large section, so quick show of hands or or two people want to read this, so we can just pause for like two or three minutes, and everyone can read through this, or do you just want to walk through this it at paste? Oh, that's! Fine by me, I just want to to feel.

C

Even if you think your opinion doesn't matter your opinion, matters you're more than welcome to speak up, you're actually invited to speak up.

C

mmm Still only can someone just say something cause a can see: video cuz, I'm, sorry.

F

Yeah I can hear you like I, don't know, I think like most people who, who actually that was documented sort of wasters, were wasted our time right so I don't know. Maybe.

C

F

Because people commented.

C

Yeah, then that is what I wanted to do anyway and gets it to the point but yeah, but then we just we just do what I intended to do cause no okay, so I would be happy with artex replied to to that question and maybe maybe put put in exactly this reference. That would be my personal feedback.

D

Had like very clear questions and the truth is, is like kind of the thing that that is in progress as well, but it's kind of moving things are moving towards improvement, but yet further questions. No. My point is dude like this. What.

C

I just wrote: it worked yeah Kirsten! You can also resolve the question basically and you you document what what you replied for everyone else.

C

So the next question once again about scaling out this is already done so or is it merged actually didn't check.

D

Me and Brian said look good to me, so it's like first.

B

C

Personally, I'm also happy with your play and link to so. Are there any other questions or comments regarding dissection.

C

Mm-Hm, so just for the record I read through that I was happy so.

C

Call for consensus, SiC observability is happy with the section above all agreed and in disagreement.

C

Very good next section is about usefulness in cloud native deployment and to a degree of being architectured in a cloud native style, which is one of the questions which we've also seen improvement, suggestions from the sick, I daresay yeah I read through it, I was happy with replies. Anyone else with comments rewriting this section.

C

Ok, so once again call for consensus, I would suggest, as you will probably expect, the same wording think observability is happy with the section above all agreed. Any comments. Disagreements.

C

Good next one, this is a dupe to words above that there is an affinity to housing operates and that the expectations of being a sincere project are being understood, which kind of gets a default. Yes, of course, there is quite some overlap with with Prometheus team and as such.

C

Basically that team grew along with it. So, firstly, I'm happy with that section: are there any comments or questions regarding it.

C

You very good so call for consensus once more SiC observability is happy with this section above all agreed any disagreements.

B

C

Being used in production, which is a tube to the above, so I suggest without a lot of discussion, the same call for consensus, but you can discuss the sake of Cervantes happy with this section about or agreed years now,.

C

Very good next one it's about healthiness of comatose and maintain ours. Looking at the numbers. Obviously Red Hat invests most, but there is a healthy, a healthy distribution of contributors. So, in my opinion we can make the same call for consensus. Sic observability, exactly with the section above all agreed any disagreements.

C

Very good, so next one demonstrate a substantial flow of commits and merge contributions. I would maybe suggest to also put a screenshot in, but I mean we can mentally reuse. The screenshot from above, but looking at the links before I would be happy with that.

C

Any questions or comments about about the distribution of comments and merges.

C

Okay, so called for consensus. Sic observability is happy with the section above Pollock we'd. Any disagreements.

C

Nice CDF alignment.

C

Bartok we might me, is this also a copy and paste from from the initial thing same as with cortex, that's correct. Okay, that's right!.

C

I I expected as much when I read through it, but I didn't I, didn't actually cross-check okay. So, given that CNC I've already accepted this section above what I would suggest the same cultural consensus, SiC observability is happy with the section above all agreed any disagreements.

C

Very nice, so there's a check if there is a design document and an overall design which I know.

C

Any any questions or comments regarding the design talk of.

C

So same comfort, I see someone tricked me around. If you need time just say no problem.

C

So SiC observer bird she's happy with the section above all agreed any disagreements.

C

C

Yeah cloud need use cases I mean it's kind of obvious.

C

Yeah for for basically the whole section, in fact that infrastructure management and such is outside of, in my opinion, completely acceptable and not putting any any non-primitive services carry mechanisms on top within the context of thermos. It's also fully acceptable, especially due to the overlap, which would mean that, if Thanos needed anything more, the project itself would be more than capable of implementing anything missing with increases upstream and then basically and use it also completely fine.

C

So same call for consensus. Any questions regarding this section, any objections, if not and again, please speak up because I keep talking. No one else is talking.

C

Good call, Consensus sic observability is happy with the section above all agreed. Any disagreements.

C

Very nice failure modes and, if they're being understood, I read through it, I'm happy with with everything which which I saw, but this is actually one of the most interesting sections. So again, if you have any comments, any concerns, if you want to take some time to actually read through it, it's all fine you're, more than welcome to just just say so, and.

C

Okay, then call for, as per the usual SiC observability is happy with the section above all agreed any disagreements.

E

I'm happy with the level of detail personally.

C

Yeah me too, it's I like how how its structured I like how it's being thought about.

E

But I think it's probably out of scope for the specific question, but I'm curious. If there is either are these things that are tested for in terms of regression but but again, I, don't I, don't want to torpedo or anything like. What's the correlation between these failure modes? What's something that would be caught by automation, I.

D

Mean please torpedo things like this is how we can learn and get have some feedback like yeah I can, since she say that we try to test against those definitely and especially in terms of regressions. We immediately kind of create the test if we found some certain back. That is happening that we fix just to make sure we're not kind of repeating that pattern now for this I think for especially for those distributed system fail overs it. It is more.

D

There is little area for improvement to get more testing, and particularly, we are thinking about the receive component which is replicated, and you know everything has to be reliable- that if one note is down, the replication will recover. You know, and- and and this this note will be joined again if it's up and for those things we recently think about, you know kind of have some Jetson style testing. Are you aware of jettison this like a nice project for that fault injection in different areas?

D

So we are definitely thinking to kind of expand on this and yeah I. Don't think it's it's well done in in any kind of project. In CNCs right so definitely some room for improvement here, but we try our best.

F

Yeah I just add to that. Like I, remember, we already brought a bunch of like tests which tests timeouts and things like that. Basically other weird failure conditions, so we have definitely have some amount of tests that are like I, don't think it's obviously hundred percent but yeah.

C

F

Try to do our best.

C

Mattresses I mean: is there anything more which you would like to see? Handle here like, for example, explicit statements about all this or something.

E

Iii, don't think personally again, no I love that this is called out in the assessment.

E

It was just more a high-level question, I think the the barrier as I understand out the barrier, but the you know to go from from incue bation on to the final phase is where I would personally expect that kind of thing to be more complete, but this is great I mean this is a deep level of analysis of what the failure modes are, and it's something that, when published, developers that want to contribute to the project can use as a starting point, and- and a lot of this is in the Tenno stocks as well.

E

I was just more curious.

E

D

A very good feedback, yeah I.

E

Mean again, this is from sandbox to incubation right, so this is. This is not like. You know the end state. Ultimately. Finally, but thank you for documenting it at this level of detail.

C

Other comments regarding this.

C

You soak call for consensus once again a suggestion. Six observability is happy with the section above all agreed any disagreements.

C

Very good next one trade-offs regarding performance, scalability, complexity, reliability, security, ET, CPP, and if those are explicit implicit, if.

C

I also read it I'm happy with it there's one comment which I have, which is relevant to the section, but not relevant to this document, which is why I didn't put in the comment and I want to talk about this after we're done for maybe 30 seconds.

C

So for the context of this document review and once again suggest, SiC observability is happy with the section above all agreed any disagreements.

C

G

Important holes um hi, um like sorry I, joined the kool-aid, but I think we should also highlight on the security side some of the trade-offs that were made I think.

F

It's my question that at like at the bottom, so we still didn't got it okay, because you made a comment. We will I think we'll talk about it later. Okay, no shoes.

A

So you're happy TOEFL.

C

C

So call for consensus: SiC observability is happy with the section above all agreed any disagreements.

C

C

Important holes, yes, Henchard's.

C

I mean this question is kind of weird cuz. Obviously the people who wrote those documents are the only ones who are qualified to actually judge the answer. So I dare say it's not a very good question, but that being said within the scope of what we can do within a document to review and I'm, not implying that there are any holes which have been tested over I'm. Just saying, I can't I, don't feel I aim to do due diligence on this beyond trusting the people involved yet called for consensus.

C

Think observability is happy with the section above all agreed any disagreements.

F

I think about whom wanted to discuss point five, so we already passed that I was wrong. That's right! That's fine! Okay!.

C

Yeah you mean in this section which, which is forward referencing yeah.

F

Number five year on was here: security right. You.

C

Mean that this can also be part of the question form for the below yeah. Okay, and here is no specific reference. We.

F

Want to ask something about this right. Yes, so.

G

Mike, like for multi-tenancy I, like it's not baked into the project, it's a additional thing. I was just curious if it's possible for users to abuse that I mean.

F

Abuse that it's a bit hard because we rely on proxies right. So basically, if you can abuse trick proxy into doing something, then yes like like now. If your proxy is behaving well, which is outside of our scope, then now basically parks ban you from like sending traffic to Thomas right. So that would be like really hard. Unless, like I, don't know it's I, guess you got some kind of you know we don't do like query, parsing or some kind of like you know, basically like sequel injection type of things, which I think it's really.

G

Hard right and like I know, we want to move it to prometheus community, but do you also are looking into moving it to Thanos itself, because, if, like I think you should claim, multi-tenancy has a core competency of Thanos as well and having it part of the Tejanos organization helps that, instead of it being in a different organization,.

F

Yeah I think, if use community organization, whether it should be in Tanis I, guess we need to discuss it with other containers. Thanks for the suggestion now put it on our agenda, yeah.

D

There there are very beautiful, yeah and I think we're we're definitely moving it somewhere, but I think the plan was for me to use community. Would that be okay, I think it's still kind of controlled kind of by from you system, I guess we.

G

Like I I think the idea is that we should move it away from that and claim multi-tenancy to be a core competency of tano's as well. That's just the feedback, because this is the first time I'm, seeing that a no-stress multi-tenancy, while reading a doc yeah.

D

Yeah but like in fact, you can make profuse multi-tenant as well. Things of this and I agree it's well technically less secure than in process authentication and like multi-tenancy, so you need to kind of secure your container, your you know, runtime environment. So definitely that's a that's some trade-off here that maybe we should point but definitely doable as well. I want to mention also that CN CF offers ask some security out it. So hopefully we can can't have another kind of very deep look on that.

G

Yep and look I'm happy with this, like what's in the dog I, just wanted to point that out and discuss that yeah.

C

Chanting back towards the context of the of the.

C

Explicit document review by adding more stuff and making things even better and more explicit I, don't think we need to revisit previous sections and can just continue. But if anyone disagrees, please speak up.

B

C

So next one called quality. Well, it's part X, so maybe we should just not test them now.

D

What project wrote its own code style for golang? That has some official ones, so I think that's kind of.

D

C

Spoiled by the pro style guide, you know Tom hates me for saying this, but it's a really nice, oh yeah I know you have a style guide in there like it. So, within the context of this document, I would go to just the same call for consensus that SiC observability is happy with section all agreed. Any disagreements.

C

We looked external dependencies and other dependencies if they are being justified.

C

I actually I thought I had put a comment in about about stating that you depend on rendering in prometheus stuff, which is totally fine, but I thought I just thought it's missing in here, because this is a significant part of the Thanos package as it were, but beyond that, and you can obviously discuss what dependencies oh I thought I wrote it in probably it's open in some comment box where I didn't click Submit.

C

Yet within the context of this, so maybe at this order, can you do that yeah? It doesn't need to be now course again. It's just edit more but I. Personally I thought it was missing that that you rely in large part on on a distinct project, which normally is this thing, even though it will never die verge of so same call for consensus, sick observer, please have sick. Observability is happy with the section above all agreed any disagreements.

C

Maybe talk about rendering in stuff. That's part of the development also happens in a different project, but you're part of that project with.

C

Really small emerging scream everyone who's following.

C

So I would also suggest call for consensus. Sick observability is happy with the section above all agreed any disagreements.

C

See ICD status I know that running CI jobs and they're usually green met. Would you like to see something about regression testing in the section.

E

Again, to go from from sandbox to incubation to be out of I, don't know that we would want to say you know full regression. Suites are our part of the minimum criteria, I'm apologize for the noise in the background.

E

Yes, it's not person again. My question before was more just having been away for a number of weeks and not doing enough homework prior to this call. I.

E

Mean certainly if we want to articulate in that section, what isn't is not there? That's fine, but because this is all on github anyway, I, don't think I don't want to block anything.

C

So for everyone, obviously in this college, if you're happy with with with what is in the document and if anything, which was interestingly, this question needs to go into the document. No, it's the time to say so. I.

E

Had one additional question about CI that I also had for cortex and that's and I'm, assuming it's the case, but you know other folks can replicate all of the CI results. I mean it.

E

Everybody seems to be using that, but I haven't looked at that on specifically like if I won't, you know again, or they make Forks of this and running with my own kind of actions in my own circle, CI account I'm, assuming that that's possible to do it's not hidden away or buried in a private way and other ways, architected that that's not doable. It's.

F

Definitely doable. One thing is that we test against, for example, Google Cloud buckets and AWS bucket, so you would have to actually set up like projects and buckets in those cloud providers other than that you can either skip those tests or just like a copy-paste yeah and.

E

The the motivation for the question to, and maybe again optionally we could add it, but you know if there are I, for example, we're running with cortex, not Thanos today or will be soon anyway in production, but I'm more than happy for projects that we're using and in my setting to contribute test results back or maybe maybe I'm running some hardware or some storage that's different than what's tested upstream.

E

If there were a mechanism for me to report those results back to the community, I would love to do so and and and just in the spirit of like accepting contributions, not just for code but also for testing cycles. You know there could be lots of folks in the community that they wanted to do that so again, I, don't I, don't think this is anything that needs to hold up moving into incubation, but as feedback for the project.

E

If, if there's not a happy path to do that, consider it because I'm sure folks might contribute right. You can find more issues.

D

E

D

We have everything they need to make file so, which is just one make file comment, however, yeah some documentation on the details: how to configure with external object storage might be yeah Wharf thing.

E

Yeah I've found in the past that complexity, the work is more on. What do you do with all the results? I could and again I want to put a barrier to entry to get into incubation to solve that problem. I think it's CN CF wide. You know if we could, if we could come up with some way to do that or see if other SIG's have already sorted this.

F

Mean like performance results or like just best results that they pass or I.

E

More and more meant performance results like you know, for folks running down our set production, there's a various topology, topologies and architectures, and you know everything from yeah I don't want to take up a lot of time, but I would expect that the capacity of the community at large contribute back for those willing to do so.

E

You know, for example, and again this is about Cortexiphan O's, but as we roll to production with with cortex, we'll have a whole separate staging environment that replicates our production environment running ahead right, so that might be a different set of workloads and stuff that I'll have test results for that I'd be happy to put back to the community so yeah, but again I. Don't want to I've already taken up too much time, because for this document, I'm happy with.

C

So called for consensus: SiC observability is happy with the section above all agreed.

C

Very good licensing, it's apache2, sic observable he's happy with the section, above all of it, kinda obvious, yet wanted to make it explicit, recommended operational models. That's also fine, in my opinion, that section but again speak up. If not called for consensus sake. Observability is happy with the section above all agreed any disagreements.

C

Okay, so there's Q&A with the project more or less I read through it. I was happy and I think this is also copy and paste of something which we had before correct.

E

Oh I'm, sorry, my audio list and I had a distraction on the previous one with how we operate it. I see that there's JSON it is that the is that the recommended way to deploy those or is there helm or other mechanisms as well or are those kind of treated outside that then I was project and are like I could go, look I'm listening, there's a helmet art somewhere is that is that not sanctioned or maintained or.

F

So yeah, right now, home chaps are basically community driven and I think there are like two or three people which are currently collaborating and basically working on to match our into tennis community people. None of anything ours actually use home. So that's why we don't have home charts right now, basically maintainer supportive, but we will shot. We have one which, which is in like tano's community organization, and there people will be able to actually use basically community maintained home repository. Also there is customized versions and yeah.

D

F

Lot of stuff written by community.

D

F

D

Like essentially, we want to be focused on the project itself and, of course, as maintenance, we have, you know kind of preferred way of deployment, but deployment model is not something that you know. The projects you recommend is rather what you already use on the infrastructure.

D

What you already have your best, your best practice right, provillus use, customized, we use JSON it, and sometimes you know operator, because we get to know separators as well from different kind of people, so I I want to kind of make sure we are on the same page that we are trying to move this outside into community, however help and make sure it's. It has best practices, but we want to collaborate with the community instead of having this very not controlled by us.

D

E

E

That's also consistent with a bunch of other projects that we've been finding right. I, don't have a huge personal love for hell. Myself look.

C

Sorry Richard yeah ours, nor is just looking at the timely, specified 40 minutes for the due diligence. We are beyond that time, but we are almost done with the document. So I would suggest we try our seesaw.

C

So the questionnaire I was happy with with what was in there. Are there any concerns? Any comments regarding this questionnaire beyond what Micah put in or not and I know, it's quite quite a chunk. We can also take some time to read it if people want to, but then please just just states also we know else personally. I was I was happy with, with the whole thing.

C

Okay, I, don't know if Michaels on the call else should me as the sick require a few more levity steps. Or are we happy with I.

F

Feel I can like add the darker pulse or like yeah stuff, like that we don't have anything else. I think.

D

Michael just wanted to add some kind of example: customers sizes of the deployments, so I think this is something we can definitely do, but yeah I don't think we required that from from cortex as well, but yeah don't.

C

Think so, okay, so I mean stuff like yeah I mean I, don't see answering this with more stuff as a half requirement within the scope of this call of agreeing with with the document. But more steps are always nice, even if it's just for sheer vanity, so yeah I need to copy this same call for consensus.

E

G

E

Concur with the previous yeah, this is great I, think I'm happy with it in its current state.

C

Good good, good good, so call for consensus on that section. Stick observability is happy with the section above all read.

E

Are there any that aren't I know that we folks don't have to speak and then things, but we really do want to be inclusive of divergent views like if there are folks that have concerns. Please, like we heartily, welcome the center or alternate opinions and views so.

C

Most of the time people don't speak up, which is kind of a pity, because I really would like people to speak up more, but just say: okay, well, don't be surprised if.

C

So final call of consensus for this document seek observability accepts and is happy with the document, as at home, all agreed any disagreements.

E

I'd like to extend, like you, know again a warm thank you for the authors that put all this together. You know the level of detail is great. You know it's I think considered me it's a good document and it's a good example like the cortex one was on on this and then and again, I. Think having this at this phase will make subsequent phases easier.

E

I'm fine having a slightly higher bar than some of the other six on some of this, because this feels to me like someone new to the project that says: hey West, a nose can actually start at this diligence document and kind of very quickly have a good, well-rounded understanding to be a contributor which, to me, is one of the most important things about this, and this TOC can also spend you know a not crazy amount of time to quickly be able to evaluate it and have a starting point. That's informed so Thanks.

C

Yep I can also I can only second, this I like the level of detail and care which has been put into those documents, technical review, so very good. We.

E

Also uh before word before, we're done just well this so time. If you haven't actually put your attendance and oh I, see most people about anything everyone, I.

G

Didn't add company so please add your companies and affiliations yep good point.

D

So do we have two minutes for last topic on the agenda? Yes, alright! So let's just quickly mention what this is about and we can expand on this and and talk more like in two weeks. Essentially the idea, if they click the link, is as on the mailing list, and the idea is that we are missing the connection between the kind of metric data into some kind of analytic use case right, so that we we have lots of nice solutions for gathering them for storing long term storage in long term storage.

D

You know those metrics, we have court externals other projects and we want to make use analytic use of this data right and and maybe switch from more monitoring. Real-Time monitoring, alerting use cases to also include maybe some hola, and you know, business analytics and intelligence I think there is a missing link and it's super hard for those people for data engineers to gather efficiently and discover the data that we have in metrics. So as country music, ecosystem I would love to explore.

D

You know possibilities to integrate this data into analytic form, so I'm looking for you to collaborate and for us together. So if you have any analytics system that you are using right now that we would love to connect with your metric data, for example, of any other source, just please mention on on, for example, on the mailing list or or on the ticket, also join us on slack since EF analytics, I, just created kind of the channel we kind of are starting to have some technical discussions.

D

What API should look like what are their requirements and I think we can talk more about that in the next protocol. But there are some nice questions that we are looking for our to answer how we can move that forward, and second, is that- is it realistic, observability concept, I.

E

Think, given the Charter that we all kind of contributed to that, certainly in scope, resistance activities, I, look forward to talking in the future, but we're running a number of analytics and a lot backends and we're doing specifically this we've started and we have some plans. In our case. It's a snowflake, my sequel and click house, but but yeah I know we're pretty short on time now. But does anyone else have an opinion? I mean I. Think I can double check.

E

What I think we called out this scenario, and this topic in in the Charter explicitly I was in scope, yeah.

D

I think so yes, but we still had some concerns on the email mailing list, but I think we stated that.

H

Yeah I would not be concerned with like territorial claims it's just like if people are interested in working on something that's just like working like who cares what what like, if there is some Odyssey written, it's not a concern.

C

There should be some synchronization, but I agree that I, don't see concerned about us doing this. On the contrary, I have been yapping about this for, like five years now,.

D

Amazing, so let's, let's have that on the next next meeting agenda and in free time please in offline, add more feedback. If you have any.

C

Good point about.

E

Okay yeah: this is relevant broadly as well. Yeah I mean.

C

Okay, okay I actually have a hard cut, so any other topics or are we done if not see you on the mailing list, on slack and in two weeks? Thank.

D

You thank you reaching for leading this. It was super efficient.

C

Thank you, bye.

C

C