keptn Community, 12 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keptn Community Meeting - November 12th, 2020

Description

Meeting notes: https://docs.google.com/document/d/1y7a6uaN8fwFJ7IRnvtxSfgz-OGFq6u7bKN6F7NDxKPg/edit#heading=h.8lt0hf8unbf9

November 12, 2020
Diary - what happened last week?
Continued working on shipyard-controller
Keptn Community Meetings now included in official CNCF calendar: https://www.cncf.io/calendar/
Recording of webinars on our YouTube channel
Building advanced SLIs with PromLens: https://www.youtube.com/watch?v=f9tRg2DWh48
Evaluating the resiliency of your microservices with LitmusChaos: https://www.youtube.com/watch?v=aa5SzQmv4EQ
KubeCon coming up! Office hours of Keptn (check out the schedule)
Wednesday, November 18 at 5pm CET / 11am ET / 8am PT
Thursday, November 19 at 4pm CET / 10m ET / 7am PT
Are you speaking about Keptn? - let us know and get some swag!
Noteworthy Issues & PRs (Review)
For each PR, please tell us if you updated the docs, or in case it was not necessary please say so.
Ermin
https://github.com/keptn/keptn/pull/2657 - Allow to load more than 20 services/projects in bridge
Florian:
Updated Keptn-contrib Services to use CloudEvents 1.0:
Dynatrace-service
Dynatrace-sli-service
Prometheus-service
Prometheus-sli-service
Unleash-service
Dynatrace-service: Adapt to next-gen Keptn events: https://github.com/keptn-contrib/dynatrace-service/issues/203
Updated nginx image used for Keptn: https://github.com/keptn/keptn/issues/2653
Fixed memory leak in statistics-service: https://github.com/keptn-sandbox/statistics-service/issues/9
Changed response format of statistics-service: https://github.com/keptn-sandbox/statistics-service/issues/8
Johannes:
(ready for review) Explain requirements for execution plane services: https://github.com/keptn/keptn.github.io/pull/665
(WIP) Formulate spec for Keptn CloudEvents for reserved delivery/remediation tasks and implement dedicated types in the go-utils: https://github.com/keptn/keptn/issues/2107
Roadmap → KEP / SPEC / Issues
KEP 06: The next generation of Shipyard: https://github.com/keptn/enhancement-proposals/pull/6
Work in progress:
https://github.com/keptn/keptn/projects/29
Backlog Refinement / Next Sprint:
https://github.com/issues?q=is%3Aopen+is%3Aissue+user%3Akeptn+user%3Akeptn-contrib+label%3A%22next-sprint%2
Planning Board: https://github.com/keptn/keptn/projects/29

Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh
Join us in Slack: https://slack.keptn.sh
Star us on Github: https://github.com/keptn/keptn
Follow us on Twitter: https://twitter.com/keptnProject

A

Hi everyone and welcome to this captain community meeting it's november 12th and uh we have a couple of uh things to share and then we will walk through uh the agenda items one after the other first thing I want to share. We've already said this: maybe in the last meeting please make sure to subscribe to the cncf calendar that also lists all captain community meetings, and with this you always stay up to date on the time of the community meetings and uh it's always thursday, 5 p.m, central european time, um but sometimes yeah.

A

If there is a time shift, please just make sure subscribe to this calendar and get updated on our new community links. um Then we have had two webinars this week and both of them are already available on our youtube channel. I've posted here at the links. So if you want to check them out, the first one is on uh using captain with promethos and building advanced slis, and uh you can do this with the help of the tool called prom lens, uh julius volts he's the co-founder of prometheus.

A

He was joining us in this webinar and he's explaining how to write a very uh actually very complicated or complex sli uh and with the help of prom lens uh we could. We could get this done.

A

The second webinar that is now live is or ready for you to watch uh is together with litmus chaos, and uh here we were building um the use case of evaluating the resiliency of microservices by introducing chaos in the cd pipeline, orchestrated by captain and uh then having uh the captain quality gates evaluating the resiliency by uh checking how the chaos actually affected our service level objectives.

A

We're also really excited that cubecon is coming up next week and uh we will be hosting uh two office hours. So there is no capcom virtual booth during kubecon, but we will have two office hours where you can interact or you can talk and communicate with the captain team. uh So part of the captain team will be there at these office hours. We will be doing a short presentation of captain and we will be there answering all your questions around our project.

A

uh First office hour is wednesday at 5, 00, pm, central european time, 11 a.m. Eastern time and 8 pm um pacific time and on thursday we will do everything one hour earlier, so it's 4 p.m. Central european time, 10 a.m. Eastern time, 7 a.m. Pacific time um you will receive the links uh to join this office hour via the kubecon website. I guess.

B

A

Via the schedule builder, if you are talking about captain uh somewhere, if you are integrating a captain in your in your environment, uh if you are giving uh like, if you're talking.

B

A

Somewhere publicly, if you're writing a blog or captain, please let us know reach out to us we're happy uh to send you uh some swag and uh with this these were like short updates from my side. uh With this, I'm already handing over to airmen to share some of his work.

C

Thank you very much um yeah. I plan to present a poor request today in the bridge- a rather small enhancement here, but it might be like it was in this case blocking and therefore it was uh yeah, it's uh good to know um it's about allowing to more than 20 services uh or uh projects in the bridge and uh yeah. Let's have a look at the issue that uh we had here.

C

uh The thing was that the captain's bridge was using the or is using the api to retrieve uh projects and services, and the api has a page size parameter with a default value of 20 elements here and actually the bridge was never providing this parameter here and we had not the use case to have more than 20 services or more than 20 projects in one um captain installation. Until now uh and yeah, at a certain point uh leon, one six uh seal here, um yeah had more than 20 services in one project and addressed that.

C

So now we uh provide it as a first step and uh yeah to make it um as easy, configurable and environment variable for projects page size and an environment variable for services, page size, uh which is now by default 50 and can be increased in case that you have more projects or more services in your captain. Installation.

C

Yeah, actually, that's all I wanted to present today. So if there are no questions, I'm heading over to vlogger.

D

All right, thank you. I'm in.

D

D

Okay, uh yeah: what uh what have we done this week uh so this week, I finally found some time to finally update uh some of our captain core uh contrib services in order for them to be ready for using cloud events 1.0 which we're going to be using in captain 0.8.

D

So I've managed to update the dynatrace service, the dynatrace sli service, also the two prometheus service and unleash uh service, so those are actually all of the contrib services that we're using in our integration tests currently.

D

So that should allow us to further proceed with making our current master branch stable again and continue with adapting all of our use cases to the coming version. 0.8 all right. The next pull request concerns the dynatrace service. So this was basically a follow-up issue for the update to the cloud events 1.01.

D

So, basically, since we're going to use new events in captain zero, eight with different structures and different event, types that needed to be uh taking into account uh here on the dynafrase service as well and actually what was having those new events will be, will make our lives a little bit easier here, because uh previously we had to basically infer uh certain things like the like how the workflow is going to proceed based on on a certain event.

D

So, for example here you will see that, for example, if we have a evaluation done event, we looked at the deployment strategy or the test strategy and based on those information we kind of had to infer on what's going to be happening next and in captain zero. Eight we're going to have uh more number events, uh so to speak. So the information of one event does not necessarily.

D

Tell us about the next steps that are going to be happening in the workflow, but now we have uh smaller events. For example, we now have the release.trigger and the release that finished event. So now we have a better overview of what's currently going on and we're going to be able to uh give more insight on what's happening.

D

Currently, all right are there any questions about that.

D

All right, if not, then I will continue with the next one. So this one was a rather small one.

D

So, basically yeah. Basically here I updated the version we're using for our api gateway for engine for nginx, api gateway, and here I uh basically update it to the latest version from 119.1 to 119.4 and that's also going to be included in the upcoming 073 release. uh So I've also made a per request for for that branch.

D

So that's going to be included all right and the next two requests are concerned, a statistic service, so the first one actually was a bug where christian has detected a memory leak and, as you can see here here in the resource consumption graph, the memory was continuously growing, even when the service was idle and that has been fixed in that pull request.

D

So the main cause of that was uh that at one place in the code, we called the we used the mongodb library to check the connection to the database and we didn't close the connection properly and therefore, every time it checked, the connection uh more memory was allocated and therefore the overall consumption grew over the time and after the fix, I investigated the memory usage and yeah basically monitored it for a couple of hours and, as you can see here now, the memory usage is constant for that service.

E

Also, uh as we discovered today um what was happening here uh with not being garbage collected with the connection, pointer, etc, it was related to the mongodb connection and mongodb itself did not kill the connection either, so that was kind of uh bad, because mongodb itself was having increasing memory over time. So, if that happens to you uh yeah that might be something to look at.

E

We should have an.

A

E

For this service yeah for memory usage.

D

Yeah exactly all right and then finally, the last pro request I want to show you today is also about statistics service.

D

So in the previous 0.1.1, the response format of the service was basically a key value map that basically mapped the name of the project to a structure containing information about a certain project and also the same approach was used for statistics about the services within that project, and it turned out that it was a little bit difficult to to interpret and also to yeah, to handle the payload in that way and therefore the requirement was to instead of using a key value map for this information to to use a an array containing the entries for each of those projects and yeah.

D

Basically, that has been adapted, and we already made a new release of the service with the version number zero, two zero. In fact. Actually here you see a better example of how the payload looks now so now we have an array for for the projects and also for the services within the project. We don't use key value maps anymore, but erase containing the information.

D

All right are there any questions about that, if not, I will hand over to johannes all.

F

F

Sharing my screen, actually, I don't have that much to show. I just want to point out that this pier is ready for review it's about explaining the requirements of a service that is running in the execution plane.

F

If you have some time, um please check out this pull request and each feedback is is welcome and then the the other issue I'm working on is still formulating the spec for the captain cloud events that will be then supported by the new shipyard format.

F

I already started working on this issue and provided the first draft, but to get this work finished, I want to wait until we have a running captain 0.8 version just to double check whether the implementation and the specification is aligned, in other words, to also figure out whether the implementation implementation need to be adapted, because we want to follow another approach, but the rm4.

F

I have to wait a little bit and keep this uh issue still on hold, not that long, but as soon as the new, the master branch is building uh the kept no.8 version. I will continue working on that one all right, then, let's jump over to the items that are in the backlog for or here in this in this kanban board um airman. I just uh recognized that there are a couple of issues that you did not uh mention in your review.

C

Yeah, I actually just mentioned the uh new feature, but haven't uh mentioned the bug fixes uh which are included in our 073 release and yeah, and we still have yeah and we have also two old features that have been presented last week. The bridge um show all core use cases on integrations page and I think then the one below so they are now also merged and.

F

This one this one has been presented last time and.

E

We don't need to present back um pull requests here um unless we really really have the time, and there is something valuable in there. I'd say for today. We are running quite well in time, so if there is something valuable to share go ahead and if not, then let's just move them.

C

Yeah, actually, the bug fixes are rather small, um yeah details like the compared with information, the first one uh where we differentiate between um compared with last past and last past or warning and all last evaluations. This depends on the parameter in the slo file, include results with score, and this parameter could have been set either using uh quotes or also without quotes and when it was provided without quotes. The lighthouse service interpreted this value correctly, but the bridge was not reading it correctly and not showing the details in this case um yeah.

C

That's it now. It also parses this correctly and shows that information correctly. uh The next one is the evaluation time frame. So uh for an evaluation. We showed a time frame how long the evaluation uh took, for example, uh five minutes zero seconds, and the issue here was that when you clicked through the evaluations in the heat map, the time frame was not updating so the times the start and the end time were correctly shown for each selected element, but the time frame was not recalculated here and uh yeah. That's now also fixed.

C

um So you always have the time frame for the selected element here perfect and the next one is about the start. Evaluation example in the integrations page, um which has a parameter as a time parameter, and this time parameter is generated on client side. So you see your current time there and, while generating this time, we also considered the time zone in which the client was, but the captain installation is using utc time for uh yeah applying the the evaluation here and yeah. This is not changed, so we show the utc time.

F

Great cool thanks for for your feedback on on those bugs and for for going over the changes.

F

All right, then, what we have in the ready for a review column are two pull requests that need a review. um Florian is waiting for that, and then we have two issues from christian. um Can I move it over.

E

Yeah, uh the fully qualified domain name, uh documentation, update, is already live and there is not much to talk about. I've talked about it in the last few sessions and the migration from travistie.org to dot com um has been finished. Essentially, um I haven't seen any reports that anything has failed or is not working, so I guess this is done.

F

F

All right, then, this column is cleaned up. um Just one quick note on the release 0.73 um issue there we met already good progress. um Well, the spec has not has not been updated, we'll still stick with uh 0.1.6 version of the captainspec. The examples has been changed and they are.

E

Released is this with the spec right, because we did introduce a new cloud event for invalidating. Didn't we did we did we do it differently.

F

That's a good point: let's, let's look at this offline, okay! Well, um let's discuss this offline, but.

E

It's not a biggie but yeah.

F

No because actually it's just an update on the release, notes, because just the release notes reference the used specification, but still uh that's true, and I think that this one has not been added to the captain. Spec this new invalidation, a new cloud.

E

Event, I think we forgot that yeah. That's.

F

F

Thanks for bringing that up, we need to do this before we go out with the new release, then I just want to remove those checks.

E

So the twist github is painfully slow. If you want to click multiple things.

F

Just go on I'll: do it in the background? Okay thanks well, but what we have done is releasing the examples with version.73. There was no need to release the kubernetes utils. They still still are the old ones. The same is true for the go utils.

F

The captain, contrib services has have also not been updated, except of the statistics service, as florian already mentioned, and showed us and in the release process of captain itself there we are also making good progress.

F

um Quite a lot of tasks are already finished and actually also, we started with the preparation in the testing phase and the all the tests also showed that uh 0.73 is working as we expect to work.

F

um Long story short. uh We are making progress and I hope this week not sure whether we can release it tomorrow, but um by monday or tuesday. Next week, uh 0.73 will be out of the door.

F

All right, yeah, then those that are in progress we want to discuss. um That's that's, okay, the issues that are assigned and committed I will also skip. This brings me now to the point to jump over to the backlog, and here I have the first thanks christian for.

C

Adding this one.

F

Just in time, just in time, it's about updating the specific, the spec uh as a follow-up issue on uh on the cleanup of the quality gate. Evaluation result. um We need to provide this new cloud event. Also in the specification yeah, that's pretty much, it is someone up there who wants to to touch on the docks or not on the docks on this on the spec.

F

I can do that cool. Thank you very much.

F

F

Then the next issue is about the remediation use case in captain odot eight, as we are now getting closer to a point to execute and run the use cases end to end. We also want to get the remediation use case working again, but there is still a couple of things missing.

F

um Yeah florian explained those missing pieces um here in this issue, for example, yeah the remediation strategy, no.

D

So, basically, uh here in this function, I've referenced here, for example, the remediation service checks, whether a remediation is actually enabled for a certain project or a certain stage by checking the shipyard. But now we don't have this remediation strategy anymore.

D

So now it uh needs to do this check check another way, and my proposal would be to check if there is a remediation.gml file for that service in the page, because that basically tells us whether the remediation has been activated all right and then also, for example, we, the service previously handled evaluation, done events which we don't have anymore, so those have been replaced by evaluation.finished events, so that needs uh to be taken into consideration and also when triggering a new evaluation.

D

uh Instead of sending a start evaluation event, it needs to center evaluation.triggered event and yeah, and also correlating and incoming evaluation that finished event. Two, a remediation workflow also works a little bit different. I've outlined it here in this description, so yeah.

D

So, for example, we can use this get remediations by context, function to check whether and evaluation belongs to our mediation, workflow and, of course, also the integration tests uh will need to be updated.

D

uh Maybe a comment about the second one, because I've looked over it today already and this one actually uses the same new artifact feature of the cli. So we need that to work before this test will run through so there's a little bit of dependency to the artifact delivery use case, but the first one, uh the test self-healing dot. This h script, which actually checks the execution of the workflow by the remediation service.

D

That should be possible without any other dependencies.

F

Okay and for the second script, which dependency do we have or to which.

D

Issue we already have an issue for that. uh I think it's the adaptation of the new artifact command for the cli, I'm not sure if it's in this backlog already.

F

No, it's not, but just to to understand the context. Let me just open it it's here in the milestone 0.8 and I assume you're talking about that one. That's.

B

Your like command.

F

Send new artifact, okay, all right yeah, then this one. um Actually, let me just copy this link.

F

And I will add it as a node that we know that there is a dependency.

F

All right, cool.

F

Then um yeah, who is who wants to work on this issue in the next sprint? That would.

D

F

All right, thanks, flo.

F

Then I have here one issue that is related to um to the mongodb, actually more, it's more a developer task, because right now it's not possible to remote debug. The mongodb data store blocking item is that the liveness probe um does not allow us to do. The debugging burned um raised this issue and it's kind of more like a chore and just to make sure that we can continue developing the mongodb data store.

E

If nobody takes it, I will leave it open and either burnt or myself we'll work on it next week burnt isn't here today, but I'm sure we'll have time next week for this okay.

F

Well then, I will leave it where it is all right. Then. Let me just briefly open this uh meter issue, because here keeping track on the updates uh that happened for the services in the contributing sandbox um organization and yeah, we can see thanks to florian, that a lot of work has already been uh done. The dynadress service has been updated.

F

I think this one is okay, um yeah almost done, then also the cloud events for the dynasty. Sli service has been uh updated and the same is true for the promesa service and I think, or as we have seen in the backlog or in the in the kanban board. A couple of pr's are currently working for a review um so that we then also can check those elements.

D

The unleash service uh one has already been merged. We can cool thanks. Then.

F

E

One as well, this one, uh the issue is still open, although can we close it.

F

Let's just double check: uh this is the issue. Well, it was not linked with a pull request.

E

Yeah then, if it's done we'll.

F

E

E

Github logic sometimes and.

F

Sometimes it works and.

E

Sometimes it doesn't, I think it picks up if you say, closes and then issue number or fixes issue number. It picks that up. But if you just say implemented issue number or I worked on issue number so and so it doesn't pick it up. Are there reserved keywords for.

F

I I think so: okay, all right yeah, but um the cool thing is that, with all those updates, we are now at the point where we can um make our travis build green again, because those services are used in the testing script and they help us to make a build successful.

F

All the others are just supporting and not needed for testing, but still we want to get them updated as well, and I will yeah keep the progress or track the progress uh here and those issues are referenced further down. Well, it's the update on the statistics service, then also the dangerous sli service and prometheus servers. They will slightly change their behavior.

F

They will listen to or started at a triggered event and will then send out the started and finished event and the notification service, which is written in typescript, also needs to get also wants to get updated um in order to understand the new cloud events, when someone is out there who likes to write typescript, um please feel free to take a look at this issue and just tell us that you want to work on it and we will.

F

We are happy to assign it to you and then there is the argo service, which also needs to be updated to make the rcocd tutorial working.

F

All right, then, I have here four additional issues. Let me just open that one. First, it's about adding the version check um to the root command again, because due to update on the cli, this version check has been removed, but we want to bring it back in order to help the users to order to inform the users when a new version of captain is released and with this issue we want to bring in this feature again.

F

Someone up for working on this issue.

C

Yeah, it sounds not too difficult, so it would be interesting to have a look on that one. For me,.

F

Okay, um yeah, absolutely cool, um it's it's cli! Let me assign it to you and then.

F

Thank you very much chairman.

E

Obviously, if you have questions as you're, um not that familiar with the go cli that we've written just ask yeah. But it's it's nice to see uh some go code from you.

C

F

And I have another actually bug that has been reported by ankit. uh He told us that um the cli does not work when using gp gpg yeah, which is a key management system on a linux and there it's not working well. uh The cli returns.

A

Usually he's also on the call, if we want to oh, if he wants to share that uh and could you you can also share uh your thoughts on this. uh If you, if you're hearing this.

B

Hello yeah so uh well, I I had a talk with chloe and I would I would pick it. uh I would pick it up in my next like once. I finish my project and I'll pick. It up. There's two issues that I have already commented. It I'll assign it to me.

F

Perfect goodness, thanks a lot, I'm kid. um Can you just leave a comment there and then I will assign it to you thanks a lot.

F

Okay, then, I just have two bugs to talk about the first. One is a bug in the bridge and the the problem is um that this label.

F

For showing this information uh at this label, the bridge is taking a look at the last 20 um triggers or root events that were that were fetched for this particular service, but in case this list does not contain a config configuration change or an evaluation done event. Then this label will not show the last deployed artifact version.

F

This happened here for for gabriel, because gabriel was using or was was playing around with with his captain, and he sent a lot of problem events to captain and, at the end, the configuration changed he went um disappeared and was not showing up in this list. As a consequence, this label was not showing the information correctly. As it's now saying, service is not deployed, but in fact the service has been deployed and the fix would be to not take a look at this list, but to fetch this information from captain, uh regardless of the events.

F

Yeah here we go actually.

C

With the pull request I presented today, this issue fades out a little bit because uh yeah we increased now the page size to 50, so the default page size actually, but still, this is an issue. If you have a many problem, events yeah in a row, it might happen that the configuration change event disappears in this.

B

C

And the api, when loading the services, I think, uh provides also the uh last configuration change event id. So the bridge could fetch that one and just show that information exactly.

F

That's true, yeah. The response of get service request already contains the configuration changed.

F

Event all right, um someone up for working on this issue in the next week.

C

Yeah you can assign it to me, of course,.

F

Thank you very much german.

F

And there, then there is another issue in the bridge, but since firefox is not that famous anymore, it's not that urgent anymore. This issue, uh the problem is in firefox when you click on this icon or in this label. That shows you the slo result or the evaluation result of an slo.

F

Then firefox does a jump and you will actually lose the focus. This is not that convenient because then you don't know or then you will end up somewhere else and yeah yeah.

C

Actually, we are showing at tooltip and in firefox something happens with the hate of this view and yeah. It jumps around and therefore you cannot see the tooltip and.

E

C

No, you don't see the details of this evolution.

E

Actually, you see it for like half a second, then it jumps it's it's mega weird. I've reported that I just verified. It's still the case, uh I'm not even sure if that's something that we can fix that easily or if it needs a contribution to barista components, yeah.

C

The question here is: if yeah maybe something is not well styled with our css, maybe some custom css properties, so yeah needs to be investigated here.

F

F

Yeah as it's a bug that has not been um solved for for the new release for the upcoming 0.73 release, I mentioned it as unknown limitations, release notes yep as it's an inconvenient situation, but still not not breaking all right. um Then this is kind of a research task first, to get an understanding why this is happening uh and then to to get an understanding also for deriving the fix.

C

Yeah, you can assign it to me, um but I would not invest too much time.

B

C

uh Yeah try some css stylings, maybe some overflow setting or hate setting might fix it uh easy.

C

But if it's not fixed uh in a easy and fast solution, uh let's say half a day, then um I would just close it as known as a known issue for now and to report it maybe to barista components and yeah.

F

Cool yeah: let's, let's continue working uh in this way, don't invest that much time, maybe just a quick research um and then, let's see how to continue.

F

Cool a kid already left a comment. This gives me another chance to assign it to you. Thank you very much.

F

Yeah- and this brings me to the bottom of this column- no more issues left, um then I'm also at the end of my my task for today. Let me just go back to our agenda.

F

This is what we did right now, then um you can do we have an additional announcement to say or are we I think.

A

F

A

For um for for this again so next week there is kubecon and we are hosting two office.

B

A

uh I put the uh agenda or actually the time of these office hours. I put it also in this meeting notes document. um If you have any questions, um please put them in the chat, so we can still answer them. Otherwise, join us in our captain slack during the week to also ask your questions, but I think we're.

A

A

So no more questions coming in uh then we can close this meeting a little bit earlier than usual thanks everyone for joining and see you all next week, bye.

C

We'll see you bye.