Cloud Native Computing Foundation App Delivery Special Interest Group, 2 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF App Delivery 2020-09-02

Description

CNCF App Delivery 2020-09-02

A

Hey baby, we want to.

A

A

B

Hello, um this is lee jong and uh welcome to joining us in the c-gap delivery community meeting and uh we cancelled the previous meeting due to the uh kubicon europe, because our folks are have speak there, so they don't make so we postponed the uh several presentations to this time, and so the main topic of today's meeting is about project presentation and we are very honored to have two projects here.

B

The first one is litmus chaos, which is already a since project, sponsored by c-gap delivery, and today we are very happy to invite them to give a project update uh on little scales. So what's new here and what's the progress of their extensive onboarding and uh we'll see new features and also what's the next roadmap. So this is the first item we have.

B

The second item we have is, we are very happy to have a flagger project here, which is a automatic, progressive application delivery system and which can work seamlessly with git with detox systems like flux or auger cities. So we are very happy to have its maintainer here to give us the first presentation of flagger uh regarding to uh to how it works, how it designed and what scenarios can be applied to how it fits to the landscape of the same cfc gap delivery. So we're very happy to have this project here.

B

So so I I don't want to waste time here, because we do have um two project to present and present, and uh so I will let a little more skills folks to take over from here to do their presentation as they are followed by flagger. So every presentation we hope it's like uh 15 to 20 minutes, but it it's not restricted so and we also have a very short q, a after every presentation. So, okay, I'm not sure. If uh who will present the latino skills project here.

C

um Hi everyone- this is karthik, um I think umar is due to present about litmus. uh He will just be joining us a few minutes from now. Could we go on with the the next topic? I think here.

B

Okay, no problem, so let's just uh start from the flagger presentation, I I believe stenson. He should be already already in a meeting right.

D

B

Hi, nice to see you yeah cool cool, cool, okay, you can take over from here.

D

D

Hello, everyone- I am very glad to be here and present flagger a project. I've started roughly two years ago, is now under weaveworks, and so we work sponsored project.

D

Let me give you a short overview of flagger capabilities.

D

First, I'm going to talk about what is progressive delivery, so progressive delivery is, is a wider term that you know, encapsulates scanner releases, a b, testing future flagging all these techniques that have a single goal and that goal is to reduce the risk of introducing a new software version in production, and there are a couple of ingredients to achieve progressive delivery.

D

Most of them overlap with continuous delivery. Of course, first, you have to have a ca pipeline that produces immutable artifacts, um something that signals that you shouldn't be using latest image tags in production.

D

Everything should be immutable and merchant in some way, either through a git, sha or assembler tag, for example, for docker images.

D

B

D

Should be that the cd pipeline should be designed in such a way that does reconciliation.

D

Most cd pipelines right now are just reacting to some event, and only then are you know applying the state, and this is something that github's covers the way that you define your desired state and that that state is continuously reconciled. Not only when, let's say you do a git push, something can change in the cluster at some point and who is who is playing.

D

D

D

Okay, I'm gonna stop shane. uh Can you kick out the person? That's uh who's.

D

D

Okay, let me let me try again.

B

Yeah, you can share again.

D

D

Try, okay, um so let's say you have your cd pipeline ready for uh for this kind of reconsideration, then progressive delivery needs a smart routing service, something that can look at the traffic and route traffic dynamically between upstreams, based on um some properties of traffic like cookies, http, headers weights and so on. So this routing component cannot work for just.

D

Let's say the layer 4 cna implementation, you need something at uh layer, seven, hey do you know, do you know who is who is.

D

B

Yeah you could check the candies over there.

B

D

B

I think we are zoomed.

B

Bombed that is never expected.

D

Okay, I'll try to go for one one more time.

D

Okay, so when we started working on flagger, we set a couple of goals: uh yeah. Okay, I'm going to stop sharing uh nazi science is the is the worst um yeah. I think you should pass or protect.

D

D

Or I don't kick, whoever is doing it.

B

B

I can't I can't remove the participants actually, but I think there's one person, I'm not very familiar with.

B

Like sammy, can you identify yourself.

B

Okay, so I will end the meeting for now and uh please join us again: okay, okay,.

B

A

A

We had a very similar problem during kubecon as well.

A

Yeah I it just so happened that when we were presenting so back of my mind, what it's happening again is it me that's like uh causing all the starts to align very badly yeah. I don't know. I thought one. Our session was one of um one upper one but uh looks like it's continuing.

B

It was very, very bad.

A

They were just playing videos, so, let's see.

A

I think it's um if that happens, it's probably best to talk not to share anything.

B

Okay, I I I found one thing to start that, so let me try this. uh It seems that I can stay to the meeting. I mean to stop anybody to annotate the presentation. I should be able to do.

B

That, yes, so let me.

B

B

Yes, so so stefan are you still there.

A

B

Yeah, so if you, when you share a screen, you you should choose, disable attended, attendee annotation.

B

There is a stating over there.

D

uh Yeah, it says you cannot start screen share, while the other participant is sharing or something like that.

B

uh I mean when you share the screen, you can choose uh their setting, which name is uh disable attendee annotation. You can try that. So I will stop sharing my screen and you can share your screen and then choose that. That's.

D

D

Okay, let me try.

D

I have share computer sound optimize screen share for whatever that's about it, maybe after more disable attendee, annotation, okay,.

B

Did you find that.

D

D

Cool, uh where was I okay?

D

Okay, on progressive delivery, so you need immutable, artifacts, a cd pipeline that applies the desired state, some kind of l7 router. It can be an ingress controller or a service mesh.

D

Talk about that a little later and, of course, you need something that gives you observability into the network, not only for performance stats which, let's say all layer, 7 load balancers will give you something like that, like request per second error or latency, but you could also improve the observability by exposing business metrics then, and those metrics could later be used in your in your delivery pipeline.

D

um When we started flagger, we set up some goals. The major one being able to you know, give developer confidence in automating the production release like deploy on fridays. It's no issue, some something will roll back if it fails and that's something should be uh thing that runs on its own fully automated, and in order to give this kind of confidence we we decided to expose.

D

Some fields, some things inside custom resources for developers to be able to control the blast radios to define the validation process, run their own integration tests or any kind of automated testing as part of the of the deployment and for those that require uh manual approval for production releases.

D

We also added manual gating to the to the whole process.

D

Another goal is right as little young as possible. As you know, in kubernetes land everything is enamel, so we want to trim that down when you define your your delivery policy and of course we want to manage the whole process from from a git ripple through guitars.

D

So flagger is a kubernetes operator.

D

It controls traffic and allows you to decouple the deployment of an application from the actual release process and flagger implements a couple of deployment strategies that you can choose, based on your um on the type of application that you want to deploy.

D

One diploma strategy is a canary release. That's using progressive traffic shifting, and this works great for applications that are exposing http, apis or grpc apis.

D

Another diploma strategy is a b testing, and this this is used for user facing apps that need the session affinity. I'm going to explain a little why this is needed. Let's say if you want to do a canon release with with uh traffic shifting for for front-end app, then what what the traffic shifting means is?

D

You set up a percentage of your users and you redirect those to the new version, but if your app has static assets like let's say, javascript or html, and also the http api or grpc api on the backend, if you don't pin a user to a specific version, then they can get. Let's say the javascript asset for from version one and the html form version two. So we can see how these two cannot uh work, fine together.

D

So um for uh for this type of apps, you have to have a way to pin users to a specific version and that's possible through http, headers or cookies, by defining a regex or or a value that matches those.

D

Another strategy is blue-green with traffic mirroring.

D

What this does is users are interacting with let's say version one and all that traffic is uh cloned and uh sent to version two, but the response is not returned to the user, so the user is is not aware that only all all their actions are basically duplicated on on the two versions, and this works great for either content apis. If you are using uh traffic metering, for I know, something that writes to a database makes a transaction, then you'll get duplicated data and so on.

D

So this works great for things like a machine learning model that you want to test or something like cache, processing and stuff like that and finally, blue green, the classical blue green with with traffic switches, a traffic switch where you spin up version two, you run the integration tests. uh You run a load test on it. You determine that that version is okay. Then you switch the whole traffic at once from v1 to v2, and this works great for I know stateful application and legacy apps.

D

How the canary deployment strategy works, so you have, let's say a deployment running in your cluster. That deployment is at version one. You apply a change to that diploma. Let's say you change the image stack to version two and what flagger does, instead of letting kubernetes just do the rolling uh update of of that deployment.

D

It spins up version two as a different deployment and um slowly starts to route traffic towards it, and while it does that it also measures, uh latency error rate and and other things to determine. If the new version um is um is okay respects your kpis, and if that happens, then it it does the in the final step it does a kubernetes rolling update on the old version once that that is finished, it waits for all the traffic to go back uh to that deployment.

D

Then it uh scales to zero the the canary release, um a b testing works, the same just the fact that, instead of using a traffic weight here, we segment we use a user segment and only those users, let's say that have an insider's, cookie or a header, are redirected to version 2 and based on that traffic.

D

The decision is taken to promote version, 2 or roll it back and blue green uh is the same, but without any um production traffic. Here you just run your conformance tests load tests. Everything goes okay, you do the switch, then your uh users will be interacting with the new version.

D

Okay, so how you you set up one of those strategies. There is a custom resource called canary inside the canary. You can set a bunch of things. You can tell flagger where the deployment is. What was the deployment name?

D

If that deployment has a horizontal pod, auto scaler or not, you define what ports your application exposes and and then you can define how the canary analysis should be run, and here there are a bunch of things that you can specify like metrics alerts, web hooks and header matching, if you are doing a b testing and so on.

D

So, based on this definition, flagger generates a bunch of objects and if you let's say if you want to do a canary setup manually, you'll have to duplicate all your definitions. You'll have to have let's say two deployment definitions, two horizontal port, auto scalers, uh two cluster ip services and so on. Then you have to add a service mesh object, objects or ingress controllers for each for each version.

D

If you are using flagger and the canary definition, then you can specify only your deployment and your horizontal port, auto scaler and flagger will generate for all these objects, including kubernetes cluster ips service mesh objects. If you are using a service, mesh or ingress objects, if you are using an ingress controller.

D

So this is how flagger simplifies a lot uh the setup of a canary release, and it also allows you to move from one service mesh implementation to another or from one ingress controller to another without having to change anything inside your your deployments.

D

In terms of traffic management, so flagger works with the with a couple of service mesh implementations, istio, link, rd and app mesh.

D

um If you are not using a service message, and if you only want to do canon releases for your apps that are exposed outside the cluster, then you can use flagger with an ingress controller like contour blue engine x and two weeks ago. Skipper got also integrated into.

D

Flagger how the validation process works, so flagger comes with two built-in metrics based on prometheus. So if you install istio or linkedin or any any kind of service measure ingress controller, these proxies will expose two metrics. One is the request. Success rate from all your requests: how much? How um what's the percentage of uh of errors? Let's say: 500 errors and um request latency, you can determine let's say in the last minute- was the average request, duration of your users and, based on these two metrics, you can set up kpis.

D

You can also define custom, metrics and we'll see. I have an example further on and you can also specify webhooks that will be calling into your integration testing platform. You can run load, testing and so on. During the analysis and of course, flagger looks at the kubernetes objects and you know, queries the the deployment health status.

D

That's based on you know, leafness and readiness pros.

D

So for metrics you can define custom metrics. So besides these two, the request success rate and request latency. If you want to extend the metrics to something else, let's say your application exposes some custom prompt use, metrics or you want to use other things that your service mesh exposes. You can create an object, called the metric template, and here you can define a prometus query or a data.query or a cloudwatch and flagger will call into all these services.

D

uh We'll run the query and based on the result, we'll uh we'll decide to move forward with the canary uh release or roll it back.

D

In terms of alerting, flagger implements four allen providers, slack microsoft, team discord, rocket chat and you can configure for a canary release more than one provider. Let's say your sre team is using slack on a particular channel and your dev team maybe is using a discord or something else. So with flagger you can. You can funnel all these events to the to the right team, no matter what what the provider they use.

D

In terms of testing, there is a webhook section section where you can define.

D

You can configure flagger to call into your services at different stages in during the calendar release. For example, you can run helm test before you expose the new version to the live traffic to your users and flagger will be calling into a helm test service will run your your hand, tests. If those are successful, then it goes to the next stage of the the camera release called rollout, and during that stage, for example, you can start a load test, while load test is needed because let's say you can deploy at any point in time.

D

But maybe you don't have live traffic, so a load test is there to generate traffic, so flagger can uh have flagger, can see metrics and and decide what to do and for load testing there are three implementations: hey is for uh is a load test for http.

D

um There is also one for grpc and for conformance testing. There is support for hand test bash tests, but you can also implement your own uh test runner and dell flagger to fall into that.

D

In terms of manual gating, um there are um several gates that you can set during the analysis. For example, uh let's say you, you push a change in your production system, but you don't want that change to be automatically deployed or tested. So there is a confirmed, rollout, uh webhook and flagger will ask you, hey I've detected a change.

D

Do you want me to start the analysis or not and after the analysis is over, it can also ask you hey: do you want to do the final stage, the promotion or any at any point in time? During the analysis you can tell flagger to roll back, even if there are no errors and all these things happen through through webhooks.

D

uh In terms of integration, um flagger being a crb controller works great with uh in a githubs pipeline. So let's say you change something in your deployment. Spec you commit that to git, then a githubs operator like flux, argo, cd or jenkins x, will reconcile that object on the cluster flagger detects the new version and starts the canary analysis for you.

D

Some things for the roadmap, maybe you've, heard microsoft, create a new service. Mesh implementation called open service mesh that is based on the smi spec flagger implements smi spec for link rd, so it should be fairly easy to extend support to open service mesh.

D

We are looking at adding more pro metric providers. For example. Maybe you want to look at stackdriver or influx db for for custom metrics, I'm pretty happy about kubernetes ingress, v2 api. It has all the things. Flagger needs to implement all the strategies. So at some point I'm guessing the ingress controllers will be switching from v1 to v2 and I want flagger to be to be ready for that switch.

D

Other things on the roadmap are using kubernetes jobs for conformance testing and having a dedicated service for manual gating.

D

And there are a couple of workshops for each service mesh implementation and here are the links to the docs and and the repo any questions.

B

Yeah I I have, I would say well quite concept, a very wonderful presentation. uh I am very interested in the roadmap of the flagger, because there are several things uh I want to ask. So uh do you have? I know you have already integrated with smi. So does that mean I can now use, for example, microsoft, open service mesh directly with flagger, or we still need some work to make that happen.

D

So I smi has different versions: linker d is on v1 alpha, one open service mesh is will be on v1 alpha 3.. There are breaking changes between apis.

D

So what what's what's going to happen is flagger has to implement every version. Then, when you install flagger for open service mesh, you have to tell it hey, use alpha 3. As the version I see, yeah.

B

So so that means uh the first thing we need to do. If you want to support uh osa is we need to upgrade or we need to make lager support the latest version of smi right, yeah, okay, they're also another question in checkbox: uh it's about uh flagger uh compared to argo rollout. Can you explain a little bit about that.

D

Yeah, the um argo rollout is very different from what flagger does flagger works at um deployment level. Argo road works at replica set level, so it will it's a implementation over the replica set.

D

So that's that's very different in terms of how you how you define all these all these things, for example, in argo rollers, you cannot use your deployments. You have to change from deployments to rollout or, however, you call it.

D

One of the reasons we made flagger reference deployment the the same way and horizontal scalar reference at diploma and doesn't come with the whole deployment spec inside of it is because we uh at the beginning, we wanted flagger to be able to take over um applications while they're running in production and I've seen a lot of users doing that by just you know, changing all the deployments back, all the charts, all the hearing, charts that, maybe you don't even control, is not wasn't an option for us so that I think that's that's one of the the main differences.

B

C

Course there are.

D

Lots of differences between them, but that's that's the main one in terms of flagger like like the horizontal polar scale, reference, something it doesn't contain, all the the actual definition of your app the deployment, the service and so on.

D

Let me look at.

D

Other questions.

B

Okay, I think uh that is what we have uh for flagger folks and again, thank you very much for joining the meeting and do the presentation and very happy to see uh what's next for flagger and uh very exciting about this role. That especially you mentioned the ingress version two in english v2.

B

I I personally really think it's uh it's really a big change for the community, because, right now we have so many ingress controllers, and this really makes uh integration with such kind of thing it's uh very hard, so I'm also trying to looking at what is the direction of english v2 going to now we're happy that vlogger have it's in its own roadmap?

B

Okay, so we're pretty much here for the first project presentation and next I will have a litmus scales folks uh to do a upgrade update for their project, especially after it has been donated to cncf as part of the same set project. So what's next most current status and any new features be sure to be aware of um so, let's, okay, so um please take over from all.

A

Right yeah, thank you harry. uh Let me first figure out uh the annotation, distable and indian addition.

B

A

Perfect, so the zoom wise we are safe. Now, okay, that was a great presentation from stephen uh I'm going to look it up, uh especially you know litmus really operates in the in that space. um So what I thought is uh really give a quick presentation uh on um what did we achieve in the last three months, um so I'll probably show a few actual uh updates?

A

uh Not really, I don't have a detailed slides, um but uh this is just a 10 to 15 minutes update, so our missions to first of all, we started using sandbox uh logo. So that's uh great. uh I think the community is really um growing after we got into uh sandbox. There are more people trying out. um So that's definitely a great news, a lot of value being added for the project itself, um so thank you.

A

uh Cigar delivery leads who have done a detailed due diligence law, so our mission statement continues to be um finding weaknesses in the implementation of either kubernetes platform or applications that are deployed. Resiliency reliability is of foremost importance for um both these personas and our little bit medium to longer. Term mission statement also is to help chaos become a very easy to use for developers as well. It is an extension to the development uh process or within the ci pipelines, but right now we're really trying to concentrate our road map short term to help sres.

A

Do chaos end to end and get some validation of their existing operations or setups so contributions? My data continues to be uh the prime sponsor, but uh the great news is that there is a lot of community embracement happening.

A

Intuit has uploaded their work on aws with litmus and amazon itself is pitching in one of the contributors from amazon is helping with docs and ringcentral is helping with helm, chart management so we're. Actually we had a problem of you know how do we deal with so many various types of contributors?

A

So we did take some inspiration from six, but I'll talk about it and one of the primary uh assets of this project itself is hub. That's uh we continue to get appreciation from having this all these experiments in the central place. So in the last three months um we really grow um grow. The project uh usage by almost hundred percent we were about fifty thousand experiments, uh runs uh when we submitted the project to it to a sandbox as an application.

A

uh Within few months. Actually, the usage is doubled and we, of course, added uh more experiments um from the community. So I just put one slide where I will use this as a guiding slide and then go to a couple of blocks and other stuff to show.

A

um I am not talking much about what our own team did. Rather, what community did and how the project is actually growing inside the community? Those are the related uh updates. So, first of all, uh uh we have now a dedicated community manager to help with community questions and to run better slack communication um etc, being available to run more meetups to go present litmus and other meetups. So he's here as well, so welcome to the project between we added uh four projects.

A

uh Sorry experiments and uh there are many more in the making and one of the good things that happened was um octatop project integration.

A

um So uh octato is a namespace uh provider for developers, kubernetes environment at m space level, so um they have taken litmus and provided it is an option to introduce chaos so that actually opened up a very good new persona where we always needed. uh Kind of crds need to be installed, so we needed admin privileges to run litmus, but this requirement gave up that strong requirement and uh gave gave an opportunity for us to rethink uh the personas. So I'll probably uh talk about.

B

A

So that's a great update and we also have um new website uh which of course is according to the guidelines and along with the new website website. We also got a contribution from an open source contributor, a new mascot or chaos. So that was a surprise.

A

So this is an sre, a chaos, bird trying to do experiments, so the community was pretty much gigged about it.

B

A

A minor thing, but it was pretty awesome that this came from the community, um so we also updated a chaos hub to with some more easy to use features um uh apart from easy to search. uh There were uh one thing that community wanted: your experiments are great, but um I need to add uh or tweak the experiments after uh so for that I need to download. Is there a way your hub can provide integrated editor?

A

So all these experiments uh can now be updated on that local cache in the browser um you can just update them, tune them and then apply them. So this is.

B

A

For making the usage of experiments pretty easy, you have copy paste, replace the regular ml editor, but it's available, although it's small technical enhancement, it makes it the user experience much more easier.

A

You know this was another thing that requested by community and then we had it, um and we also got uh very organically uh engineers from azure test team and then someone from rancho community they've tested all the experiments uh of ligma's on that platform and sent vrs. That hey. Can you add this as certified platforms? So that was a great addition.

A

And then we also got a container solutions, have done a good study on open source chaos, engineering tools- and um this was a good information for me as well on our other projects. So both the cnc projects are covered uh litmus and chaos mesh. They both are coming almost with the same feature scoring, but it's great to see that chaos engineering is being covered as part of the cncf uh encouragement.

A

Both the projects are scoring pretty high. So that was one good thing for the community and other thing. We run monthly meetings and community meetings. We wanted to mentor some teams who are coming in in different areas, so we took the inspiration from six cncf for kubernetes six, so I think some other projects are also doing uh machery, for example.

A

um So we have done. uh We have created some six, the concept of six within litmus and uh we try to have uh once in a month or on demand basis at the same meetings recently the doc sick, as well as the deployment six. Some meetings have happened, so how we do that is, we did not create any projects, but we made use of.

A

um If you go to the teams or in the litmus cares organization, so we basically defined uh our teams, um and these are uh you know whoever wants to come, participate and they're bad as part of that- and this is like a pretty simple way of telling the community that you know there are multiple groups- you can choose observability or deployments or integrations or documentation wherever you are interested, um it looks like chaos is applicable to almost all areas or across the scenes of landscape. So there's good amount of interest that is coming in.

A

So this is one way where we're able to segregate the big interest into smaller groups so that more contributors can come and drive themselves. The idea is, somebody will find themselves as a major contributor or driver within that group, and then they will add the product roadmap prioritization list to that entire project. So that's another thing that we started. um That's received pretty well from some parts of the community and uh the other one. um We are getting a a lot of new queries.

A

um Air gap support was requested from you know somebody and from community. I think uh shantanu who's here. So we added some additional help. There arm support was another uh top request. We added that as well.

B

A

In terms of a roadmap, the our team has been very busy in developing a litmus portal um which I'll show it's not it released, but alpha is going to come out later this month, um but the idea of utmost portal is um it's not just about experiments for kubernetes resources or applications, but sra should be able to create or orchestrate complex chaos workflows um in order to find the deep level um weaknesses in their operational deployments.

A

So a good portal is needed. uh Easy to use monitoring is very, very important and what's happening to my chaos workflow, so a lot of work is going on and we have embedded the entire argo workflow into the project.

A

uh What I mean is um uh argo: experim workflow uh wraps all these experiments and then we call that as chaos workload, but it just gets done in a declarative way, which has got a ui support in the front end as well, and we've been getting a lot of inbound interest uh in terms of we need graphana dashboards. uh Chaos is great, it's very easy to use. But how do I monitor right so and um there are a lot of uh existing experiments.

A

We got more updates as well as more queries uh our intention to help in terms of can we have more granular tunables within the chaos hub. So this is that's probably the update, and uh these are some of the. There are dashboards that are in the making uh they'll go about in this release. We have monthly release cadence. The whole idea is um prometheus, matrix and graphene.

A

There's a lot wealth of information, um but when something happens you can easily know, uh but with um chaos litmus. You are willfully introducing these faults, so you need to know what was that problem? Was it during chaos, injection or did it naturally occur and uh we needed to know um or we needed to give that perspective of uh willful fault uh simulated fault versus the organic one. um So this is an example.

A

If you are taking a cpu node, you can see that this chaos was introduced, a node chaos most likely, and you can see that node increase. So it's um it's. uh It's also a good once it is introduced. There was some issues with that. Node the node utilization was not coming down, so it could. You could say that we found an issue there, but a lot of things can be um interpreted as a weakness or you know as a strength. So this is an example of a well-known microservices demo application from weworks softshop.

A

So we took that and we added a bit of chaos, metrics uh and chaos, uh uh graphs to that uh area. Graphs- and this is actually coming- uh contributed or the process of being contributed by uh the community, and uh they are also coming in and doing um a lot of.

B

A

uh Developments uh to the sick, observability area, so that's really heartening to see and the other thing that I think we just started. We probably need to work with captain team uh allies and other things. We did not find a lot of bandwidth, but we did research captain and how uh it actually, both captain team approached us, and then there were some other community members asking for you're able to do chaos. um But how do I interpret the results?

A

Monitoring is great, but we needed to see if we are really when a fault was introduced. Did some slos fall off right?

A

So that's the idea so we're trying to not use all features of captain, but just the quality gates, feature they're very nice features so end users can use both captain and um uh litmus together, um just like litmus and workflow. So our go of clothes were used to create chaos, workflows from chaos. Experiments captain can be used um to integrate quality gates as well. So we see that you know a lot of cncf projects are coming together becoming more solutions. We are also trying to focus uh if we are.

B

Integrating it.

A

Is better be um a cnc, a project so that way people um not only adopt, but there is more community engagement as well.

A

um The other thing that we are dealing with right now is multi-tenancy looks like kubernetes has introduced um good features in in with respect to multi-tenancy, and we also did uh some small uh survey, our research uh among various forums. um The somebody here can give more feedback, but at least what we got to know is many team members using a single kubernetes cluster is a not an uncommon scenario, so what it really means that they would expect whatever the kubernetes environment that I would need for development, I will.

A

um I will have it uh on a given kubernetes cluster within a namespace, um so we did bank on crds heavily the entire architecture, so crds need.

A

Privileges admin privileges, so though we have now introduced admin mode and name space mode, and there is something called standard mode. Also, the whole idea is, we are working towards making litmus usable by end developers or sres who may not have admin privileges so that it's it's pretty easy um to use, and uh there are some thoughts about uh developing config map based uh approach rather than crds.

A

But easy option is let at least the admin use. The crds install the crds and later developers or other people can come and use litmus within the imp namespaces. So that's another thing: there's still a lot of discussion is going on in the community, but that's I thought is an interesting update to share so overall, I think we are progressing very well getting a lot of support from different different uh communities within cncf.

A

So thank you very much and we're looking forward to getting a little bit more advice from various projects as we get a little bit more popular. So that's all I have, um but I'm happy to take any questions.

B

Well, thank you very much um and I'm very happy to see the progress that the uh little scales made after joining the since and we're happy to see that there is a positive feedback from the community and the growth of the usage of this project. So we now have don't have enough a lot of time for questions, but I'd like to check. If anyone has any questions, so we can let uma.

B

A

All right, I will post that slides uh the meeting notes.

B

Yes, thank you. Yeah yeah, please link these slides uh to the community meeting notes, so other folks who cannot join meeting, can still check uh what's going on in the meeting what you guys presented over there so well really appreciate it for uh the presentation from uma and stanford about they will try to engage more people and to present their awesome project, regardless of whether they are part of things, if or not, or they have any motivation to donate it or not.

B

We are trying to make this thing a more uh collaborative and open community for people to talk to each other and share knowledge about application, delivery and application development. So I hope that we will have more projects present in the upcoming meetings and, uh if you guys have any idea or any project you want to see- and I will have it very happy to talk with you and I will try to reach out to the project as much as I can to have.

B

You know these these awesome ideas present in the uh meeting okay, so I think we are pretty much uh uh our a little appreciation for today's meeting and uh again we really thank you for everybody joining this meeting and uh I will be super looking forward to see you uh focusing you next time so bye-bye.

B

Thank you bye-bye. Thank you, yeah. Thank you very much.