keptn GSoC in Keptn, 27 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keptn mid-term GSoC Demos - Integrations for GitOps-based Deployment Tools: ArgoCD and Flux

Description

Google Summer of Code (GSoC) is a global, online program focused on bringing new contributors into open source software development. GSoC contributors work with an open source organization on a summer programming project under the guidance of mentors. GSoC contributors accepted into the program work on projects to improve or enhance the Keptn project and its ecosystem. During the mid-term evaluation our mentees will present their projects. In this session Meha Bhalodiya, a GSoC contributors in Keptn, will present her project: "Integrations for GitOps-based Deployment Tools: ArgoCD and Flux".

To be able to use GitOps tools in combination with Keptn. The main goal of this project is to find a proper way how the tools could interact with Keptn and to build an integration for such tools. Deployment tools can be integrated via the helm-service or via the job-executor service. Currently, there’s no way to integrate GitOps Tools in Keptn. This project aims to provide a way such that such tools can be integrated into Keptn return deployment states after the state has been synced.

More about Keptn in GSoC and the projects: https://github.com/keptn/community/blob/main/mentorship/gsoc/2022/projects/gitops-deployment-tools-integration/README.md

A

And just connect to youtube so during the introductory slide, uh so will you be screen sharing and or do you want me to screen share.

B

Okay, so, like your part, you can share your screen and when I will start presenting my project demo, then I will share my screen.

A

Okay works for me.

C

A

A

D

How is everybody, okay, good? uh I need some more luck out. There.

A

Okay, so let's begin in one minute: now: let's let others john and brett return back.

C

C

So should we begin then.

C

A

Let's see how many people join through backhand, I see viewers on youtube but yeah homo versions from through meetup picture. They should land here on attendees.

A

D

Do you want me to.

A

D

Video and off presenter.

A

No, it's perfectly fine just stay and I will definitely ask you to share a few words after the presentation. Okay, sure, okay. So let's go.

A

Hello thanks for joining us today uh we are starting a obligation, presentations in the capital community. This year we have four projects and we have will be representing her project today.

A

So just quick organizational information, we are broadcasting to youtube. You are free to join there. We have a chat, um so you can just ask your questions for that or you can go to the meetup page and join us through webin. So this way we meet with other presenters and everyone can join there too.

A

A

Could you please share the slides so that I'll do quick introduction and then we go through the rest of the presentation.

B

Or do you want me to share the slide.

B

Yes, you can share.

A

Oh okay, so I will just screenshare that.

A

A

This is my screen.

A

B

A

Quick introduction, uh so uh this year, captain participates in google summer of code. Google summer, of course, is global event, targeting open source internships this year, it's available not just to students, but also to more experienced developers who want to join open source and learn how to work with open source communities.

A

Google summer of code is hosted by google, open source team, basically open source program office. uh This is 17th or 18th year of the program, and until now there were more than 10 thousand of participants.

A

This year, captain uh participates at the first time. We have four projects. This year we received more than 20 applications from potential contributors. We had 13 mentors who were interested in different projects and we had very cool, currently eight project ids.

A

So, first of all, thanks to all contributors and applicants, as always, we cannot accept all the projects, but we hope to have for some additional projects as a part of some lfx mentorship program.

A

So this year uh we selected four projects so first off always git ropes uh in captain with integrations with flux and argo cd. This is the presentation today then, tomorrow we will have a presentation by eric about the entity crds for captain again for guitar series and rajiv will show the new documentation engine powered by docus, which we're going to adopt in captain and last but not least. uh Next week, uh jainamma will show the key six integration service he's working now on together with adam gardner and pepe khanna from k6.

A

So this is our agenda. If you're interested in gsoc and captain in general, we have a channel captain slug. We have a number of projects which you can find on our website.

A

So here you can see information about the projects, timeline, all information for mentors and contributors and if you go to a project page so, for example, today we are talking about this project. Here you can find some details and references so that you can get more information about the projects. You can also see that many projects are being held in uh the independent channels. So, for example, there is gitobs greater working group for captain and if you're interested in today's presentation, you can go there.

A

C

A

We have a youtube playlist, where you can find all the project we also trying to broadcast when possible and we host meetups on captain community, so every project has their own office hours, but we also invite all you all mentees to join developer and community meetings and present their statuses here. So, for example, tomorrow we have a developer meeting then community meeting next week.

A

This is also a venue where you can discuss projects share your feedback or, if you're, interested in mentoring on your own you're, also welcome to join these meetings and talk to us, okay and yeah regarding the schedule. So, as I said, there are three sessions all these three sessions are listed on my cncf community here. So this this session is tomorrow's one and one on thursday.

A

If you're interested to participate just go to the community, cfio slash, captain community uh go to any projects, and here you can click the sleep button so that you join the session and get a reminder. I'm looking forward to meeting you there.

A

And I guess that's it from me, so uh I would like to invite her who would present her project and then we will have a wider discussion with all the participants.

A

So the floor is yes.

B

Sure, thank you uh is my screen visible.

A

B

Okay, so hello, everyone, my name is mia. I am uh the captain g soft contributor for integration for github's base deployment tools focused on flux and argo city this summer. So I am honored to give the midterm presentation today about this project. I like to give okay, sorry, okay, I'd like to give a brief introduction about myself.

B

First, so I am an undergraduate student from gujarat technological university uh currently pursuing a bachelor's in computer engineering and yes, I am a newbie in captain open source community, but yes, uh gsoc made me more involved in this captain project, and so now I am a bit of familiar with this project and I am grateful that captain can give me the opportunity to realize the idea in this project.

B

So yes, let's see like first of all, I will say what is captain so essentially it it's quite hard to understand what captain is when you first look at it. So it's an even driven orchestration engine for cloud native apps like it started around 2018.

B

It uses a declarative approach to automate delivery and operations, so they can be scaled up to a large number of services.

B

Also, captain evaluates sl os, like the service level indicators and it provides a dashboard alerts and auto remediation for them.

B

It also provides integrations with the cloud native ecosystems, including prometheus argo city, cloud events backstage and many other cloud native projects, so additionally, it has many different capabilities of bringing your own tools to it and then connecting your tools in a decoupled way. So we are really liked it for the quality gates. Because then, when we wanted progressive delivery through environments, we could ping say prometheus and say: is this app okay? So yes, similarly to the argo rollout and then the quality gate would let you pass and go into the next environment.

B

So you can also do the things like gravitation as well.

B

Now, uh what's the motivation behind starting this captain integration? So yes, as we know that many people use kitops tools and could profit from the lifecycle, orchestration features which captain provides. The resources are intended to integrate argo cd in captain workflows, whether it is argo, cd or flux. So yes, using this tools as a deployment, we can get a better overview of what's running in our system and how it differs from your desired state.

B

So yes, using as captain as an orchestrator tool, we are able to integrate this with testing tools which can do slo based evaluations and implement automatic remediation actions.

B

So the main idea of on which we work around currently is the integration is mainly based on notifications on the argo, cd flux, side and web hooks on the captain side. So argo cd itself is used as a change detector in the git repository of its app now when a change is detected, it notifies captain about that and captain can trigger a deployment after approval or automatically after argo. Cd has finished its work. It sends a notification to captain and further workflow can proceed now, as you can see in the image provided in the slide.

B

Initially, there might be some configuration of argo, cd or flux somewhere. Now, let's see like, I will uh explain the stepwise. So the first step is argo. Cd or flux is connected to git repo, which contains deployment manifest or say, helm, charts and it syncs with this repo now second step. What it does is when there's some things new argo city of flux triggers the deployment and notifies captain about this like uh using deployment.triggered or started event now.

B

What's next third step is the manifest or health charge applied to the target cluster and, at last argo, cd or flux notifies captain that everything is finished?

B

Hope you uh get the idea of this project now, let's have a little background on argo city like you can use uh argo, ocd notifications for slack teams and they have the concepts of service template and trigger, so the trigger will come in from argo cd and then it will whatever based on your template. So our template was just a cloud event and then that would be sent to captain to continue orchestrating the business logic. So we use once like out of sync is quite good because our goal becomes out of sync.

B

Then we can get captain to go, say, okay, go see. If this is going to work in captain it works like uh a concept of shipyard file, so you might not get it the first time, but yes, a shipyard file is uh basically consisting of three things. So what it is like it has a stage, a sequence and a task so similarly to event based architect- and you know your micro services and your apps, which are bringing into ci cd.

B

So I can send a cloud event and say: okay, go trigger this stage and then it will go through my task and my services will be subscribed to them.

B

Now, let's have a look of this: both get ops tools, workflow. First of all, we will go through argo cd, workflow like what and how it happens inside. So yes, we can understand it step wise. So, first, what it does is argo goes out of sync and triggers the captain out of sync notification.

B

This is a triggered event for a delivery. Captain sequence now, what's next, the second step might be captain web hook listens for a task event of delivered dot trigger, which comes from the other, which sends a web hook to the argo sync now the web hook will automatically generate the deliver, dot started task cloud event, and now, when the app is synced and healthy argo runs, the captain deployed notification template which sends the deliver.finished task cloud event to captain.

B

So finally, captain then progresses to the next task in the sequence, if there is one, for example, a load test or quality gate check and many such events.

B

So yes, this is this was the basic uh overview of the how argo cd works. So, let's now move on with the flux workflow.

B

So what flux is focused on like we can get input or source changes from hand, release updates data form, get repositories customize or all of them at once, and the notification controller here is the main component in the flux essentially, which gets the notification and then flux will send the json payload, which can be changed anytime.

B

So image says at all now there are two main concepts in notification: controller that is alert and provider. So let's have an overview of this to get ops toolkit components. So, first of all, let's have a look at provider, so the provider type can be slack.

B

Mstms discord, generic google chat and many such now when the type is generic specified. The notification controller will post the incoming event in json format to the webhook address. As you can see in the image we have specified type generic under specs section. So this way you can create custom handlers that can store the events in elasticsearch cloud watch, track, driver, etc.

B

So to now uh to receive the events as notifications, a provider needs to be set up first and once you have a setup of the provider you can. You need to create a new alert resource in the flux system to start receiving notifications about hand release.

B

So, let's move on to see what alert is now to verify that the alert has been acknowledged by the notification controller. We need to test with the command cube, ctl, hyphen and flux system.

B

So now what alert will do, but there will be multiple alerts which we can use to send the notifications to different channels or select organizations.

B

As you can see in the image event, severity can be set to info or error. So here we have uh used info now when it is set to info. The controller will alert in some cases like if a kubernetes object was created, updated or deleted.

B

If health checks are passing or whether c is delaying the execution or we have somewhere, error is occurring and when the severity is set to error, the customized controller will alert on the error encounter during the reconciliation process. This includes customized build and validation errors. It applies errors and health check failures.

B

So yes, if you don't specify the event resources namespace in the image we haven't uh specified the namespace. So what happens is the alert namespace will be used.

B

So yes, this was a brief about my project, like what I have used, what I have done in the phase, one of the gsoc so far, and yes, some of the things which I have learned is like how captain can orchestrate the argo deployments, how captain and argo can work together and something about services like I have learned about job executive services.

B

I have implemented it in the captain bridge and I have also gone through some web hooks integration services, like uh still, we are working with the slack webhook integration and, yes, I have uh learned to set up many new techstick on my local system, like setting up flux with your cluster study about hand, charts ci, cd pipelines and yes, so far. I got to learn many new things while working in this project. So thanks to the captain community and yes at last, I would like to uh give the idea to the new contributors.

B

If they are stepping into this project, like you, can all feel free to give your feedback on our captains like channel and yes, the report links uh are shared in this slide and yes, we are not yet done with 100 with our project, but, yes, you can try it out and feel free to open up the issues, and we are more than welcome with the enhancements any new features and if any improvement is needed, feel free to open up the issues and work around them. So, yes, thank you.

B

You can connect with me if you have any questions or just have a quick chat. So, yes, thank you.

A

Thanks for the presentation, there is a question from kunal in the chat. He is curious. Why not use customize as a tool for managing kubernetes configs instead of hand? Charts is a specific needs for kubernetes, or can other tools be used.

D

Yeah, maybe but john, when you answer that one sorry scope of this project, you can use anything so you can use both helm, charts and customize. So the beauty of the integration is that you can either use, and you know some people like to use customize. Some people like choose helms, like I, for example, use customize home together.

D

um So the idea of what we're building here is that whatever you want to use, you can bring your own. So if you like customize and bring it um it's totally up to you, so there's no right or wrong answer there.

A

Okay and yeah, so I wanted to ask: uh what are your plans for the next stage of the project, so my understanding that now you basically have two demos for flags and argo cd. I'm not sure are you going to show them today, maybe next time.

B

Yes, so so phase two will be more focused on testing with the different scenarios.

B

And yes, I will try to document all the things which we have worked on.

A

Understood and from your perspective, for what flow is better for end users, is it argo or flux and what tool was easier to integrate with.

B

Liv is like the easier integration was the argo cd, because it gives many easier functionalities like it, provides the ui and a far easier vendor as compared to flux.

A

Understood, thank you.

A

So if you have any questions, you can uh just ask in the chat, et cetera and then, while I will ask mentors brad and thomas to share their perspective about the project, how it's gone and just whatever you want right, would you like to go for it.

D

Yeah sure, sorry, um yes, congratulations! Mia! It's been really good. So far, um you've learned so much. You know from it. It's a lot of things to learn at once. You know: go lane kubernetes flux get up. You know, there's so much, but you've done really well and um yeah. So so far the the project's been a good success and we've got a lot of work to go for the next phase. Two as well.

B

Thank you for your feedback.

C

Thomas, so for me, it's the same thing, so um I think especially building an integration for flux is a really hard thing in this case, um because it didn't um it, it was kind of different than the than the first approaches we had with argo, because you can you couldn't hook in into the into the into the process because um flux by itself it applies whatever is in the git repository. So um therefore, this was pretty nice. This was. um It was pretty nice that you uh for me for me.

C

I was positively surprised that you that that uh you know the way you handled all of this and um yes, very cool from from that perspective, from the from the implementation perspective.

C

So um I think you took the same approach as I would have taken so simply intercept the event and send it and and forward it and yes, from that perspective, I'm I'm also really really um eager on what's happening next, so um what what you will do in the in the next few months and how this will evolve.

C

So congratulations to the prog to this progress until now,.

A

So maybe to channel the question from absolve opportunity to try out the project at the moment, or does it make sense to wait.

D

A bit I, I would say you can, but what we've done so far, is it's almost like a proof of concept, so what we want to do is we want to bring this logic back into flux instead of being in captain.

D

We already have too many integra too much services, so we wanted the capability to be able to send the cloud event from flux, so you can try it today, but I would recommend maybe waiting one more month and then you can sort of get like the the beta version of it instead, um but please we're always keen for people to test as well. So we to help you get it going, because we want to ensure that we're meeting all the use cases from folks in the community as well. So please reach out to us.

D

If you want to.

C

Yeah thank you and for reaching out basically github's working group on captain slack.

A

A

So any other questions or maybe additional comments.

A

So yeah, then thanks a lot for your work uh and again. If anyone is interested to know more about this project uh yeah, please join the community channels. Please join the meetings and looking forward to seeing the production, maybe not production, but at least so preview for this project. So I'm using hardware personally, I would be interested to try it out and flux tool. I have never did done anything with flags beyond hello world, so I guess it would be a good opportunity for me to actually try it out.

A

So thanks a lot and good job.

B

A

Okay, so if you have no other topics to discuss today, I think we can uh close down earlier um tomorrow. There will be a presentation well tomorrow there will be two presentations uh for g sub projects. uh So if you're interested you just go to community to the captain community- and here you can see that uh yeah there is midtown demos, uh ltcft is for captain get ups and your docs engine. So I believe it's on the morning right yeah, so it's 11 00 a.m, central european time or 9 a.m. Utc.

A

So, if you're interested just join the session and right before that, the results at developer meeting where we will be discussing what's new in captain core, there will be demos by contributors. Also, there are some topics we would like to discuss like duplication of hound, charts and the geometer service.

A

So if you're interested please join and then we can have a lot of conversations so two hours of captain tomorrow, thanks again to everyone working on jsoc this year,.

B

A

Thanks everybody thanks.