Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lifecycle of a kubectl Command: Harden Kubernetes Setup with Automation - Sanjary Rahman

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Lifecycle of a kubectl Command: Harden Kubernetes Setup with Automation - Sanjary Rahman, Booking.com

We at Booking.com run tens of on-premise multi-tenant Kubernetes clusters at scale. To automate integration with our existing bare-metal infrastructure and for running kubectl auth pipeline, we run an ecosystem using custom Kubernetes Controllers, Pod Security Policies and Kubernetes Auth & Admission Webhooks.

Kubernetes provides end users with limitless possibilities of automation to harden cluster setup, secure authentication and authorization pipelines and validate workload definition as per organization requirements which most of the users are not aware of or make use of. Most of the time hardening Kubernetes setup in a multi-tenant cluster with per namespace based setup itself can turn into a huge toil for the operators.

In this talk, you will see how we at Booking.com have achieved the aforementioned features in a fully automated fashion with zero human intervention involved.

https://sched.co/MPaH

A

Welcome everyone, I'm Sandra I'm working for booking comm in Amsterdam and.

A

Sorry, it froze.

A

Yeah, so my talk is, as you can see, life cycle of coop CTL command here, I'll show you how we hardened out kubernetes, set up with automation and how that gets reflected throughout the lifespan of a cup CTL command. So let's have a quick look at the agenda of the talk.

A

First, we will see the different multi tenant, kubernetes cluster architecture we have in booking.com and the usual deployment works, slow that our developer swallows and then we will see the challenges that we faced during managing this automation as sorta managing this multi tenant on Prem clusters and the solutions that we came up with and also we will see how we automated all the solutions for all the multi engine clusters we have across our infrastructure.

A

Then we will see how these automations are leveraged by the lifespan of a single cup serial command with an example, and if we hopefully get time, we will go through a Q&A session as well. So, let's begin so, let's have a look. First, the different cluster architecture we have so currently at this moment in booking.com, we run tens of multi-tenant kubernetes clusters where we categorize these clusters into three environments. Basically, the first one is the dev environment, where the developers actually hack into their prototypes and new ideas.

A

Then we have the staging environment which resembles the production environment, but is used mostly for testing purposes, and then we have the production clusters where the application runs instead of the production traffic. Now the we will look at the usual deployment workflow on these clusters in every environment, so the workflow is very simple. In each environment we have two types of clusters: one is application type in the application clusters. The workloads are actually run there. The pods deployment is always a workload.

A

Type of objects are created in application, application cluster and then they run there and we have a separate type of cluster called management clusters and in this management, cluster Dave's actually go and deploy their application with our in-house built.

A

Multi cluster Kennedy deployment solution shipper, which is a communities native way to deploy multiple cluster route traffic and shift capacity, and all these things, so our developers just go to management cluster and then tell shipper to deploy it across our fleet and shipper then deploys the application across multiple clusters. If you want to know more about shipper and the workflow, you can read more about shipper here now the challenges that we faced during managing these multi-tenant cluster, because, as you saw, we have different types of clusters in in our infrastructure.

A

Every cluster has to maintain a consistent, workflow permissions and configurations right in every namespace, so each cluster actually shares the same name spaces so that we can maintain the deployment workflow in a smooth way. So the challenges that we faced initially was the project management, because multi-tenant cluster means multiple service owners and multiple deploy their multiple projects in these clusters, and we need resource isolation which gets solved basically by using kubernetes namespaces, because that creates the business isolation, but then comes the resource management, because it's a challenge because different projects has different types of requirements.

A

Some requires more resources. More CPU, more memory, some require less write and we handle such cases using kubernetes native resource code object that actually acts as a safeguard so that one project accidentally does not continue, consume entire cluster resources, so resource quotas actually helped us achieve that.

A

Then we face the challenge of providing auth management or maintaining the auth management, because in all of our clusters, we want to make in the same ought protocol an autumn pipeline so that users can just go to our management cluster and apply the application without thinking they have permission in every cluster or not. That just works, because we have a consistent off pipeline across all clusters using kubernetes roll bindings and the kubernetes are back and using custom oscar books.

A

Then next challenge game is config management, different organization, a different requirement for the application to run with certain configurations right. So the way we propagate these configurations is using config maps and these config maps are injected in every namespace. Well, the namespaces are created and, finally, the validation and safeguards of the resources that are being created, and we achieve that using admission, webhooks from communities and part security policies which are also known as PSPs. Using this, we can ensure a secure environment for your workloads to run now.

A

This solutions at a glance may seem a bit simple to create for one cluster or a couple of namespaces, but when your cluster increases and when the this projects running on your clusters increases because right now, in our tens of clusters, we run almost half a thousand of services in production, and at that point it becomes more challenging or toilet for the admins to maintain those clusters, and this is exactly the point when we need automation to saw, came out to solve all the problems, because doing them by hand is not feasible by that time anymore.

A

The way we tackle that is using a word source provisioning, sorry, what space provisioning, which takes care of automating, all the solutions that I discussed in the previous slide. So the automated works is provisioning works with three main components, which is the first one is Service Catalog.

A

This is something that we built in our company and the Service Catalog is just a catalog well before deploying any new service in kubernetes or any new project project, our developers actually registered that service or the project in service catalog, so that they can maintain an ownership, information and service catalog just gives that keeps track of the projects that are created and the ownership meta information. Then the next thing is the booking I am we have. This?

A

Is the identity provider for all of our users in the company which is kept in sync with with the service catalog, so that it knows which user has permission to which project or which user is admin of which project? This is just keeping track of the different projects we have, and then we have a custom, namespace controller that actually does the heavy lifting and it does the automation for us based on the projects created or kept in or registered in service Drake service catalog, and now we will see how the workflow works.

A

So, first, our developers create a project in the service catalog and at the same point of time, our identity provider gets inked with service catalog, with the information for the ownership like this user has created the project. So this user is an admin of this project now right and then what happens after? That is the namespace controller that I talked about in the previous slide runs in every cluster.

A

We have that keeps a watch on Service Catalog and then, whenever it figures out, there is a new project created that namespace controller in the local kubernetes cluster creates these following resources. It creates a namespace for that project to be deployed on it, creates a role binding for all the users who will have access to this project. It will actually assign a group not static users in the role binding, we'll see that later it creates a resource quota object which actually acts as a safeguard for all the resources to be created in this namespace.

A

It also creates limit ranges, object, which is a kubernetes native object that defines the default CPU and memory requests and limits for every container. If the users do not provide by themselves also it propagates some or odd specific configuration to every namespace that gets created and out. The applications that get right deployed in these namespaces can make use of these afterwards.

A

Now I, we will see how the lifespan of AI or lifecycle of a cup CTL camp command actually leverages the automation provided by the previous provisioning work. So so for that, let's take an example for a simple Katia mole is just a simple busybox pod, which we will see will get created in an m-space called by Trenton, for example, and it's run the busybox image with image pool policy, if not present and for say a developer. Now we'll run cook CTL player f on this pod manifest object right. So what happened?

A

Actually, in this case, first, the user or developer runs the coop CTL command or makes a list API call to the kubernetes api server and then the coop config file that we use looks any other coop config file that everyone uses, but instead of using static, token or certificate for auth, we actually use the coop coop CTL executed angel plugin.

A

As you can see in the users field, there is no credentials there and instead they it has a command field that has a path to an executable which is supposed to generate a token for every cup CTL command. Someone runs and after making a cup CTL call command. This command fills executable, actually is invoked by coop CTL itself, internally, which we wrote and that actually generates a custom token that our identity provider understands. Then this binary outputs, this kind of JSON payload.

A

It has a kind exact attention and it has a status field with a bearer token that it creates and an expression timestamp for the token right. So you can generate any token from your own exact attention, executable, which your identity provider understands, and that's it when you do that, when this executable does that this request is sent to kubernetes api server and then after cabinet is going to accompany the passer of kubernetes api server then sends it to our custom Otway book for validating the token that is being sent.

A

So kubernetes then sends the request to our custom, auto web book and the example request. First, that kubernetes api server sends looks like this. It contains the API version, the kind that provides in the JSON payload is token review, and then this in the spec field. It injects the token that came from the executive initial plugin when the proxy tell command was run. As you can see.

A

The token has my bearer token, which was created by the the exec tential plugin here in the token field, then our customer, who actually sends this extracts this token from the token review, object and sends this to our identity provider that contains the information for all the users in our company and then identity provider, actually checks with the service directory project permissions and returns back the the projects that this user has access to and then the username also to which they took this token, belongs to.

A

And then the auth webhook sends back the response to kubernetes api server after authenticating this request and that the example response looks like this. If the request is accepted that the exam then the example response looks like this, it has an API version kind is again token review in the status field.

A

It says the value of analytic authenticated field to true, and it also sends the user meta information with username, to which this token belongs to the UID of the user and the group's the group's is actually generated by the earth web book after reading the project permission from our identity provider. So we just came up with our naming convention that the group name will look like this. The project name : the permission. It can be editor admin viewer.

A

However, you set up the role binding for your namespace will see that afterwards, after that, if the request is rejected instead of getting allowed, then the response looks like this is the same as the allowed response in the kind it will be talking review, but in the status it will be authenticated, false and the user will get back.

A

This response all the way back to the output of coop CTL command, that he ran rather user, and so this is the custom auth workflow that happens in our cluster and it happens consistently across all of our clusters. Now, how can we we saw that the how the example request and response looks like now. We will see how to configure this auth web hook with your when it is API server.

A

You just have to pass this two flags authentication token web config file where you pass a coupon thick file and you pass a flag, authentication token web cache TTL by default. The value of this TTL is two minutes. What this flag does. It sets the timeout for a decision that is taken by the the earthwork. So after that this decision is not cached anymore. So, if you are using a short loop token, I would suggest reducing it to something like five seconds or 10 seconds.

A

Then the coop config file that you pass to the first flag looks like this. It looks exactly like any other group config file, it has type config, it has clusters or any user, set a context and current context, but the difference is in the cluster server URL. Instead of passing it an API server URL, you pass it to the custom, auto a book URL that you built yourselves for your organization and you were running and that's it after that your auth wave book is configured and it can validate your user.

A

The next thing, then, after authenticating this request, this request then goes through the kubernetes are back now. How does our back knows about this request now? So, just a quick recap that in the world's best provisioning, we saw the controller create that this role binding object in the limb space right and the role binding object. Definition looks like this: it is type role binding. It says the namespace name, bike rental that we created the name of the role binding object is, is admin and the role reference is cluster role admin.

A

So it's referring to cluster role admin and then it's assigning this role to a group, not any particular user to a group called bike. Rental, : admin, ok, so dynamically, every user. Who will be a part of this bike? Rental Pollin admin group will have admin access to only to this namespace, though the cluster role here is admin, but the user will not get this permission on entire cluster. It will be on the namespace itself only because it's just a role reference and it's a role binding object, not a cluster.

A

All binding object, okay, so this bike, rental : admin group that you can see here was actually asserted dynamically on after in the earth way book when the request was sent- and this example response was returned back by asserting this bike. Rental : admin group there and that's how coupon it is. This role binding object in the namespace will authorize this request in the namespace. So this is how kubernetes are back, will allow or disallow this request based on if the user has permission in this namespace or not, then after passing through kubernetes are back.

A

This request goes through kubernetes mutating admission waiver, so kubernetes admission controller provides us a way to define mutation web hooks to intercept any requests to your kubernetes api server, and then you can change the definition of the object or inject new things there based on your organization requirement, so you can customize every payload that gets sent to your cluster.

A

Some example use case of the mutation mutation web book is, you can assign random non-root you IDs, because root UID is vulnerable and also root. Uri is injected automatically by kubernetes. If you don't define any so you can assign or change the UID to some non random UID based on the namespace in the pod. You can inject environment variables in the pod. If the pod does not contain when the ones that your organization requires, you can inject some labels on the pod before even the pod.

A

So these things will happen before the pot actually gets created in the mutation wave book. You can also inject in it containers or site cars in your pod definition, even though, if your user or developer does not define them in the pod definition. Now we will see how the example request from kubernetes api server to this mutation. Web book looks like so. The payload will look like this. It will have an OID field which will just be used for auditing purposes as a reference, and then it will have the resource type here.

A

I ran coop, CTL, create command so on the pod creation, so it will say it's a pod and then the name of the part that was in the pod definition and the namespace where the body is created on an operation which operation was made on this.

A

So it is, it was a create operation, as we saw it can be creative date or delete based on your coop CTL command, and then it also will contain the user information like which user actually made this request, the you idea of the user and also the groups this user belong to, which was provided by the earthquake that we saw in previous slides, and it will also contain the object definition that the user is requesting right.

A

So, after reading this object definition, you can actually analyze the definition and make mutations on top of it to change the definition. If you want or inject new things, and also if it's an update operation, then you will have an old object field as well, which will show you what was the previous state so that you can take a diff and based on that, you can decide and that's it. That's the example request the example response will look like this.

A

It will have an UID field which you will have to propel it from the request field so that you can cross-reference them during the auditing and then you have to pass the allowed field to be true, and you have to pass a batch type that you want to make here.

A

I'm, passing JSON patch and currently, as of now mutation will book only supports JSON patches, and then you send a byte array of all your patches and the patches look like this here underneath it will have an awful which will say what operation you want to make I said, add and I pass a path, adjacent path in the JSON definition. It's a suspect, containers 0, which is the first container and then image pool policy and the value I'm setting here is always what this patch will do in the poor definition.

A

We saw the image pool policy was, if not present, that's how I created it. But if it goes to the mutation wave after mutation, it will turn into always. So this is just one example that you can do so you can change basically any field in the per definition or any object definition that will go through the mutation web book.

A

Now we will see how to configure the mutation web book with your API server, so you have to pass this enable admission, plugins flag in your API server definition, a binary when you run it and you have to pass mutating admission web option there, that's how you can enable the mutation webbook and then you have to create a mutating webbook configuration type of type of object which will define what are the objects on which this mutation web book will be applied.

A

Okay, so here I am passing metadata part mutation and then I define the webhooks configuration where we'll pass. The client configu RL and that will contain the URL to the mutation, vapor, query or mutation web book is actually running and serving requests and it has failure policy fail. It can also be ignored, so failure policy defines if the if there is an error in server side or this way book is not it not reachable, then what will happen? Will it fail open or will it actually fail the requests?

A

So if you've set ignored, then the request will be fail open if a mutation workbook is down. If it is failed, then, if mutation, why book is down every pod creation request will fail based on the policy and you pass the name of the apartment name of the configuration and also the define the rules like which objects will be applicable for this mutation.

A

We have up here, I'm, passing resources, pods and the tree and the operation type create which will actually define that only for what creation request send the participation request to this way book and after doing coups it'll apply on this waiver configuration your pod mutation. Will book is configured in the cluster, so that's part mutation, way book or any mutation wave book that you want for your cluster for any resources, not only just pause.

A

Now, after going to mutation, the request will go through object, schema validation internally in kubernetes control in kubernetes and mission controller, because you can change any field or definition in the object right then kubernetes needs to check that if it still complies with the default schema definition of the core objects in inside communities and after this validation, this request will get forwarded to a validating admission web book. So it's quite like mutating admission, workbook but validating admission. Where book also intercepts the request, but it cannot mutate it.

A

It just validates that if the field values are correct, as per your organization requirement, you can check any field, then you can reject the request based on your requirement. If some field has some unexpected value or if someone is trying to do something differently than your expectation, some example use cases for using validation. Wave-Wave book is like you: can check legitimacy of the board or service or object that is being created for us.

A

We just check if this project, if this spot is register, I mean sorry the project this spot belongs to is registered in Service Catalog or not, then it. You can also check, like the images that your containers in the pod is used that are being used, are actually from trusted, so image registry or not, and based on these decisions, you can actually reject them. Reject this request immediately before even the pod is created.

A

You can also check the number of containers in the pod which actually we have tested and found out that large number of containers in one single pod makes some performance degradation for the daka runtime engine on the host. So you can check that and even and if the the number of containers in the pod is higher, you can reject that request, as well with proper message, then you can also check the presence of some certain labels or even environment variables.

A

If you want that are present on the under container or not on the pod, or not also, you can enforce certain based practices for the kubernetes resource declaration, like how the label case will look like you can verify them and validate them.

A

So the example requests for the mutation of a book looks exactly same as the sorry validation, where book looks exactly same as the mutation web book, but the response looks a little bit different.

A

If the request is accepted, then the response looks the following: it has the response field, it has Eid and it will have allowed will set to true, but it will not have the patch field because validation, where book cannot patch object definition, but if it is a deny, if you don't want to allow this request to be accepted, then you can send the response with this UID you say: allowed filled to be false and then in the status field, you can set a some error code with, for example, this this this time, I'm sending 400 and a custom message why this request will not be accepted right so that your user gets a nice feedback that why this pod request is being denied with proper error code and message.

A

So the the configuration for validating webhook is same as mutation workbook. You just passed validating with admission webbook value in the flag, and then you create a validating web book configuration instead of imitating web configuration object and the rest of the definition looks exactly same as a mutation. We hope you can define the rules the same way after going through validation, validating admission web book.

A

If the request is denied, you can send right from here a denial message with error code to the user, a full example, implementation of kubernetes, validating and mediating admission where book can be found in this link. If you want to go through it after passing, through validating admission web book, the request actually boast resource potter controller, which is the last admission control that happens in communities admission controller.

A

So a quick recap that our automation actually created a resource quota object in the namespace right, so based on that resource, coder controller, we'll see if it can sustain one more part in this namespace or not. If it can, then it will allow this request, but if it cannot, then it will reject the request. Saying you are out of quota. You cannot create a new board in this namespace right now, after passing through the source quota controller, after only that it gets stored in kubernetes back in storage ACD.

A

At this moment, this part now exists in kubernetes state kubernetes, HCD cluster, but that does not mean the pod will. Pod has started run running right away because after storing it in a city, then port security policy or a PSP kicks in support security policies are a cluster wide resource that manages some security, sensitive information for the containers in your pod and you can restrict different accesses from your controller eyes using part security policies.

A

So, in order for a pot to actually start it has to comply at with at least one persecutory policy that you have in your cluster. Some example use cases of for security policy. Are it can prohibit running part with UID 0? So if container breakout happens, it can so that containerized processes cannot make any harm to your host processes, because if it runs with route, ID route, EU ID and if it has privileged accesses, then it can be dangerous for you cluster.

A

Then you can allow certain or no host part at all to be mounted within your pod just to be safe and secure, so that your parts do not access the host parts without poor security policy. Actually kubernetes allows you to run any part as route which may have root access on your node, because that's the default behavior of any vanilla, kubernetes cluster. So you can restrict these things using poor security policies.

A

Other things you can do is you can forbid access to the host network namespace or host IPC namespace and host PID name spaces, because allowing this will actually allow the contraries process to have same almost same level of access to the host processes, and they will be able to sniff into other post processes or other containers running on the same nodes. They will be able to sniff their environment variable or configurations as well.

A

If they have this kind of thing permissions or even they will be able to manipulate the network stack as well on your host node. What the pod is running. You can also limit Linux capabilities, which are mostly associated with super user permissions like cap, net admin or or cap net binding, so that what you? What can happen is it may have more privileged, privileged escalation permission. Then it requires so that what it will be able to do it will be able to actually escalate privileges to its child processes.

A

I mean the containerized process will be able to escalate privileges to a child processes. More than itself, this can be limited, may limited using the port security policies so that you are safe and secure. If any container a breakout happens, some more use cases are.

A

You can allow a certain range of you, IDs and group IDs, with which your containers will run with, so that you 100% sure that it won't be able to use any other UID orgy ID, and you can also allow certain types of volumes to be used by the ports in your cluster, like you can define like only allow secret persistent volume, config map and download API, or you can also add emptier right, so only those volumes will be able will be mountable by your ports.

A

In the cluster- and you can also say, allow only a certain range of host port access in the host network namespaces, this type of things you can define- or you can also make sorry root- filesystem inside your container, to be read-only, so that container cannot write anything in your namespace. So these are the configurations that you can do you can you can allow enable the admission PSP using passing this flag and you have to create a default PSP for all the ports in your cluster.

A

You can just be at a port security policy, object with restricted name, and then you can fill the the other specs as per the as per the example shown in the link, so I'm not showing them here, and then you will have to create a cluster role and cluster all binding. Here you can see them so that the service accounts which reach your port will run will be able to use these things.

A

We use this port security policy and a cluster all binding will be able to then use this cluster role restricted and it will be applicable to system service accounts, which is all service accounts in your name space in your cluster, and then it will also get a applicable to a system authenticated which will be all authenticated. Users will be able to use this port security policy, so by default, every part will run with restricted policy.

A

But if you have a valid use case for your for having privileged spots, then you can create it by creating a privileged PSP and giving some relaxed permission to that PSP and use it only in a certain namespaces, with certain service accounts by creating additional cluster rules and class rule binding for Espie Espie, and if multiple port security policies are there only the ones, and that comes the first in the alphabetic order will be used for by upon this is tricky one, but that's how it happens for port security policies.

A

After passing, through port security policy, only then a pod starts actually running and if the user gets the final response back. This is how the entire lifecycle of the coop CTL perman looks, like you know all of our kubernetes clusters in a very consistent way, because these are all automated not done by hand some key takeaways are you can customize your workflow using custom controllers or maybe using a framework like Co builder, to automate any work so that you have in your organization? Custom controllers are very useful.

A

You can get some example information from there and you can also reuse your organization, existing organizations, existing permission, provide provider using kubernetes custom web books so that you don't have to implement something new. You can also leverage the admission controllers, our web hooks that kubernetes provides to validate and mutate the resources so that you won't. You know what is happening in your cluster and finally take the opportunity to use poor security policies to enforce a security environment for your workloads to run.

A

Thank you so much and if you have any question or if anything looks daunting to you in the slides, we are hiring you can come to me. I can ensure a smooth onboarding process. Just kidding. I will be here. You can ask more questions here. Thank you. So much.

A

The slide links are here: if you want to scan so you can find it if.

B

You do it three minutes. If anybody has a question, they want to ask sure.

C

Hi, thanks for the talk I would like to ask if there was any decision between Susan outwork hook instead of JSON web tokens, and you seen an OECD validation from.

A

C

Server so check the idea. French is not love the token instead of active validation, okay, so.

A

You're saying what? What about using OID C right.

C

Instead of instead of using an active validation, if you discussed it internally on and.

A

Why did you choose.

C

Past evaluation, instead of online offline validation, right.

A

So we are using this kind of dynamic behavior because, as we saw, every project gets registered in our Service Catalog right, so permissions actually changes dynamically. So if we do offline, then someone if a project permission changes in a namespace right, then someone will the person who will lose the access will have to read re login to get the new type of permission if we go through this dynamic works. So this is how we are actually dynamically, giving someone permission or revoking someone's permission and every cup CTL command goes to that.

A

What works slow and that's, why you saw in the roll binding we created a gave the permission to a group, not a particular user. So this is why we use this kind of dynamic behavior. Does that answer you.

B

Hello. Thank you for this talk. How much of this is open source, especially the Service, Catalog port, so.

A

The thing you saw here, except for the auto hook, because the auth wave book can talk to any identity provider that your organization has write. Rest of the things are like from directly from kubernetes. The service catalog is not open. Source is just in-house built, but it is only used for the purpose of keeping the service.

A

It has a database and interface users go there and create a project, and then the user name based on our energy provider gets stored there, but is not something like outside, because this is very tightly coupled with our odd structures so but yeah you can have any source right now. You have to include with your works, though each one.

D

A

Because the interact thing I forgot that the interact thing you saw here, the deployment solution that I mentioned that goes to multiple multiple clusters that is open, source and I, provided the link there yeah our multi cluster Canada deployment solution is fully open source yeah.

D

Yeah I have a very general question: why did you decide to give developers or give developers cube CTL at access in the first place? So why don't you go through some automated process? Only let rent Jenkins to do cubes it applies, but let the developers do it.

A

Very nice question this is kind of like third or fourth iteration of the platform as a service that we are providing right now, and we took that path that the one that you are suggesting. We came up some kind of magical tooling that we will help users to do their job, but the server does not say that the reality was not like.

A

This users are not happy at all when they go through a predefined path, because there is always like in our 20s clusters, more than thousands of developers actually work every day and everyone has their specific needs.

A

They see that kubernetes as a as an open source project, which has a lot of possibilities, but when you give them a path or a CI pipeline there, you just narrow down the road right so that then, based on our user feedbacks, the road that we took or the approach that we took here, we are providing all the features of kubernetes that kubernetes provides, but in a controlled fashion, so that users and do not feel left left out. So this is exactly why this is based on our user feedback and user requirement in our company.

A

B

And with that I think we have to stop. Thank you very much and thank you guys for the great questions and.

A

Please remember to rate them on the app. Thank you.