Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Running Istio and Kubernetes On-prem at Yahoo Scale - Suresh Visvanathan & Mrunmayi Dhume, Verizon

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Running Istio and Kubernetes On-prem at Yahoo Scale - Suresh Visvanathan & Mrunmayi Dhume, Verizon

At Yahoo!, there are 18+ production grade Kubernetes(K8s) clusters and my team operates one of the largest on-prem K8s clusters handling 150K+ containers, 500+ applications and serving 1Million+ request per second. Mission critical Applications, such as Yahoo! Sports/Finance/Home are deployed and enabled by K8s/Istio platforms. The journey started 2 years ago as a ‘proof of concept’ with K8s and signing up for ‘early engagement program’ with Istio team to adopt Istio/Envoy to modernize our stack and move towards micro service architecture. During this journey, 1.Build Identity platform which provide unique identity for workloads 2.Enabled workload with sidecar envoy proxy and integrated with in-house Custom CA & RBAC for authN/Z 3. Build tools to manage both Istio & K8s cluster at scale.This talk will detail how K8s and Istio/Envoy used to deploy/secure/connect workloads @ Yahoo Scale.

https://sched.co/Uaci

A

Thank you kiss.

A

Can you guys hear me, but afternoon come on it's after lunch come on good afternoon, thanks for joining us today, we today we're going to talk about kubernetes and sto at Yahoo and what we have done. What's the scale here running and what are challenges some of the integration we have done. So, if you look at this slide right over here, the couple of things it says, numeronius like this is early 2017.

A

Before we started this community support and East your effort, you can see another n number of stack, a number of team that has their own way of tools and things they built and duplication across right, and we try to avoid this. How can be unify this so that there's single way for us to deploy the code can dimerize the code and there is no duplication across the team out there. How are we looking today, you can see over here. Everything is containerized, but you talk about no java even read this like stateful application.

A

We are running in kubernetes today at scale and all of them are containerized and we use kubernetes as an orchestration layer and we deployed in our own prim cluster. So to just to add, you can see that yahoo sports yahoo finance all this critical application today is completely sort of kubernetes tack and also adapting kubernetes. It also gave us an option so that in the future we want to take advantage of any of the public cloud offering. We can just run the application out there without making any code change or modification to the applications.

A

So that's what we actually just to continue. This takes time. Ok, just to some numbers out there right we've been running kubernetes for last two years. We have nine hundred ninety plus application, and this allows stateless application, which is serving sports, Arriaga commerce, finance, not yahoo.com, even yahoo.com poser sword from Cabela's cluster and we are distributed across seven different data center across the world, and we are doing 1.5 million RPS at the edge.

A

That's an ingress layer and we also have stateful application, like all our key value store, Redis, which is a chair and as well as we discussed doing. All of those things runs in communities today and we have an awesome operator which manage this Redis Buster is it's a way that moving the shard and rebalancing the shard whenever the pod goes down and new part getting added.

A

So I talked about 18 production, great cluster I had from the previous things out of that. The nine of them are. Production remaining are from canary environment, where we do traffic and see how the performance and application before we roll it out the changes to production. This is just a graph on I, really like this dashboard there. It gives the federated view of like how are we doing across my production cluster, so in production today, as I was mentioning, we have how close to close to thousand application.

A

It's just one more week, we'll be hitting 1k applications and we have close to we integrated with NetApp Trident for the storage. So we have, you can see that precision volumes all the stats is completely available from the Cape federated refer now. So how does a Agrico system look like today with kubernetes right.

A

It's a very high level, so you can see those components which are in orange are those things which we have built and remaining everything. We took it from the open source wall and we are running so to your left. Yes to your left right to your left, you can see that there is an ingress controller. The ingress controllers, Apache traffic server based this is an ingress controller. We are using Apache for various reason, because it's from Yahoo it has lots of plug-in it's needed for our application to spits out.

A

So this is an ingress layer which is trying to do 1.5 million RPS and that has been integrated with kubernetes api, where it dynamically finds this endpoint and feed it to that ingress controller. So when the request comes in, it knows where the dynamically changing pod IPS are, and you can also see to your left. There is an internet request site and these are the requests from either employed portal. Are those services which are not deployed in kubernetes and trying to connect to your service in communities?

A

So those requests we use and why so, it's still managed in why and if you look at your ride as a screwdriver, a screwdriver is our CA C D platform, and this is where we put lots of effort to comply slicing so that the user can leverage without understanding and tell ecosystem of kubernetes resource and object, and everything we'll be talking about that in the upcoming slides, and you can also say something I'll Athens out there.

A

Athens is our open source platform for service identity, as well as fine-grain our back system, and we have built webhook in communities where is integrated with Athens to do service, authentication, user authentication user are back and service are back so I just talked about Athens.

A

So this open source platform- it's available. You guys want to take a look at that. You can go to this particular water, and it's just said this is service, identity, user identity and find going our back system for the dynamic infrastructure.

A

So what's in the pot right every a part which is getting deployed into our kubernetes cluster, we have three be false: ID cards right: it's mandate! You there's no way you can get out of this. So there are three side. Colors orange are logging sidecar, which takes a log file and send it to your logging infrastructure. It could be Splunk or it could be any other associates. You have and you have matrix ID card. This one is to take the matrix and send it to primitives our per se.

A

We have our one on frame monitoring solution called Yama's. We can also send the data out there and we have sia. So what is here? The service, identity, agent and service identity agents.

A

Responsibility is to make sure that give a unique identity to every workload deployed right and every part which is deployed in our platform as a very short-lived, unique identity which is provisioned through Athens, so we'll be discussing more about that and we have NY which uses that certificate for more it's for listening, as well as as a client certificate and those services which is not using sto service, miss in a more of service to service.

A

We don't have that anyone, those services which is using NY even have that enjoy as a sidecar, sorry, miss % so I talked about hey. We have 18 clusters and we are managing this cluster. How are we building this cluster? What does it mean for us right and how there are ten multi-tenancy? The way we define multi-tenancy right and today I call a soft multi-tenancy, for the reason is like there is not we will be building. We don't build a kubernetes cluster for every application out there right and we build one cluster.

A

We try to share it with all the application out there. So we call soft multi-tenancy because they are isolated through a namespace and to add to that, we also build single tenancy cluster to its based on the use case, as well as a security requirement. If our security team says that this data is very sensitive, you cannot be co-located with some other non sensitive application. In that case, we build a dedicated cluster F.

A

If not, we just share the cluster, so better utilization, better isolation and to continue like you wants to have a soft multi-tenancy, and the next thing is like we don't make the changes. Even the cluster change. There are other applications, you just don't make a change and then try to push it to across all the data center.

A

At the same time, to avoid blast radius, our push processes Maurice you can shall we do one Colo and we have metrics based promotion which will take that value and see how the changes are and then, if that satisfies all the requirements of the latency, is not high. There is no less there's no more 5xx or any other error code. Only then it promotes you, the next cluster, in a very sequential way. We deploy our changes, I talked about, say: hey. We provide this off multi-tenancy. That means like hey.

A

If we, a user from namespace da, you should not be able to be where access resource from namespace be, if say that hey keep CTL get part in the name sports namespace. If it is not part of that particular namespace, you should not get do those operations. So what we have done over here right over here, you can see that every namespace in kubernetes is mapped to an Athens, don't mind so kate s finance caters as a namespace in communities. It's mapped to a final skaters domain in Athens.

A

So you to our line, you can see to your left right on to my right. Your left, you see in the 80s I define the policy. What am I saying out here? If you look at the policy right, I'm saying that hey hello, get list create. These are a bunch of action, I'm allowing it for the resource. Quickens considered communicator spawned part exactly deployment for this resource I'm allowing this operation, but that particular roll.

A

You can see that kate has deployed as a role right now that role, whoever is member of the role you can't do this operation, so we can go. Do this fine-grain her back in Athens now, if we have in use case where hey I want to make sure that certain developer has very less access to my production instance, you can go and create in policy.

A

We said that he can only view log file and he cannot do exact or delete or any of such operation on any of the resources in production, and he can be in a separate code role called hip developer and he has very minimum access, so he can define that fine grain access with Athens and that's all we manage such a way that user name space is mapped and Athens domain and the responsibility of that a test domain admin is defining this policy.

A

We are based on his need, so I talked about user right in the previous slide. I was talking about. The user is part of this particular road perfect on time. Oh he's not going to reboot my laptop. So now, when a user is interacting, I was saying that every user has an identity right. It's a ten provisioned identity. So now I am going to interact with kubernetes api right I'm, going to say that a cube, C tailgate part and the finance namespace right I'm going to pass my certificate. This is my user certificate.

A

You can see the common name, it says Suresh. You said that Suresh P now, when I interact with the kubernetes api kubernetes api understand it can attend the gate that request, because kubernetes api runs with its own CA as well as 8 and CA, because he knows a thon CA. He can authenticate this user now that okay, fine, we are authenticated, is that enough know anyone can be authenticated because everyone can fetch and it answer to every employer out there in which an Athens identity attend certificate for themselves. The next step towards status, authorization right.

A

This is a people this hook is primary responsibility is to make sure now that I Hugh made an operation I've attending ated you. The next step is together. You are authorized to do that operation. So let's look at this right Suresh. As an user he's trying to get deployment and a finance namespace, we take the subject, access review and then what we do is we'll try to translate and forward to Athens.

A

You can see that you are at them and you can see that it's sending to Athens and expecting Athens to do the validation check against the policy and response back boolean weather allowed or denied okay, if you are, and user who's allowed to do this particular operation from the hour by definition, the policy definition I was mentioning. The response will be alone, it's not denied and we respond back saying that you attend I. Do not allow to do this operation so now that Athens is a very critical part of our system, like every interaction.

A

Every kubernetes api interaction which you try to do is goes to athens. So we try to avoid that network hub or per say athens being single point of failure for us, so we have optimized. Our web box is aware that the entire athens policy definition as a role definition as a member information. Everything is cached locally, so that the web, who can make the authorization locally without making a network hop calls. This gave us a better performance for the user and then also like hey a wider athens, being a single point of failure.

A

That point of time now that I talked about kubernetes cluster and then how the clusters are build. However, authentication model is I wish. Our attrition model is, how do we define tenancy and sought multi-tenancy our multi-tenancy next thing we want to do is like bring in hto to this world right, and why are we looking each to your day?

A

Okay, why service miss like you guys are, can read this? You can find the same definition from the hto portal.

A

Awesome feature: zero code, change right and then lots of other advanced features such as circuit breaking with me all this feature, because one of main reason is like we lift and shifter this application, which has been running in the bare metal and to containerized world right if they have to go ahead and enable and bring this kind of feature, we need to rebuild this feature into all the languages which out there. If you want to build a circuit breaking right, you need to build a library for every language out.

A

There select large of duplication right, so this was fitting our bill. When we saw that okay, I can bring this particular stack. It can I can do lots of things even even unify and consistent, be get the authentication or authorization circuit breaking all these features.

A

Instead of building libraries across many different language, x3, that's what primary skills so just to add our service, miss layer is sto and I just want to make sure that I give credit to the sto team, because in last one a half years we have been working with your team and teams from Google, but an early engagement program to bring history into Yahoo.

A

So we've been running in production a last almost as an English layer in the English player for last 100 years, and we are now pushing changes towards their service to service is also has an invite proxy as default option. So they're, not there it as I was talking previously, not all the part which, in production in kubernetes, has the invoice ID card. It's less than five percentage of the work floor. Has that one in 2020 what we like to get his secure communication, energy, Electress network, every service, the service communication.

A

You would like to make sure that is encrypted right and Stephen, and why is a default things? Because we can do in a very unified way, authentication authorization I can do it in very unified way. Right and without duplicating the piece of code across many different languages exiting so we work with them to roll it out. We test it out, and today in production we reuse ingress as an wine. Without date, five person, which are less than 5% of the workload, is running bit and were sidecar.

A

So when we talk about secured communication with each, do you can see that this is how the flow looks like so the service EFI is going to communicate to service B right. The service makes HTTP call to dependency, and the proxy in the EPI layer appears that connection to a mutual TLS, and he makes a connection to NY right and then NY senses to service be locally in HTTP con.

A

Now that we said that you want to encrypt this connection, that means NY need a certificate right, and this is where ever see a comes into picture. What's the CS job I was just mentioning. His primary job is to mint a certificate for every workload deployed out there very unique short-lived certificate. How did we do that right? So in a second.

A

We build a cool, Yahoo, skaters identity, provided this primary job is to mint a unique identity for every part out there, and it's also periodically refers to the identity right and every certificate which is issued to the service. It is based short-lived X to PI zero nine certificate. How does it work? You can see the flow out here right and you can see when a part is getting created right when the part is getting created in the node, where the cubelet is running. When it's trying to create the part we have a flex volume driver.

A

This primary job is to create the amount and created token ID, which is a unique token ID to the particular part and give that mount information to see a sidecar and what that mount exists is an UDA socket an identity, agent process which is running locally on the node. So now that the creation process is done, the startup sequence starts. So when the CF starts.

A

What he's trying to do out here is like a I, want a certificate right and he's going to take that token ID, which was monitored part of the creation process, and he makes a cut using the UDS. He makes a call to identity agent and I didn'ti agent, the guy, which is a process running in the node. It takes that one and validates is a valid token ID, because he has that information, because the flux volume driver created that locally, as well as in Silesia as a read-only mode in say this year.

A

Once he validate this valley token ID he wins. He gets additional information about the part from the cube, let the API locally. If everything happens within the box right, the equivalent API response pack, all the metadata information about the part such as his IP service account all those information. Once you have the complete metadata information you take that one I didn't take that one and he makes a calls to Athens. Now Athens got a request with the CSI asking someone to mint a certificate.

A

So what does Etta's do that records is also was attested with JWT, but Athens cannot validate the certificate, the JWT for the reason, because it doesn't have the key to validate the date ability and the letters also need to know that Who am I meeting this get fun. Is this lab valid guy there's some random guy, some of the invading the CSI with all the part, meta information and he's sending to Athens that Athens going to meet the certificate for him? It's not like that.

A

So Athens would want to make sure that is a valid guy. What is that a tenth does? Is he takes that information and he sends it to our identity provider? So you can see the identity provider what they he's. The primary job of him is like take that information validate the jet ability and he makes sure that the part meta information which was passed from cubelet and he valid against cubelet api.

A

Those information are lied once the validation is done, then he responds backs to it and saying that he's a valid guy he's a valid part, is still running in kubernetes cluster. You can mint a certificate for him, then he attempts means a certificate and he passes to either in the agent and identity agent gives to SIA and CMO gives the certificate as amount to NY. So now that NY has a certificate, it can use that certificate as a listening certificate, as well as the client certificate.

A

You know that NY has a file watch on the directory whenever the certificate changes it reloads it, and these certificates are very short-lived, not more than four or five hours, so they periodically refresh it. We can also go up grosu far now it's like four to five hours. So once you have the certificate that anyone has the certificate right and next step is to enable authentication what happens, meaning enable at the negation initiative right and NY comes with an odd filter.

A

His primary responsibility is to make sure that he validates the incoming client certificate once he validates he extract the principal from the spiffy URI and sits that principle. Why is that confer is important because, once you are attending ated, the next step towards this authorization right, so you need that principal to authorize whether the user who's connecting you are. You, sir, are the service connecting to you. It's authorized to do the job or not.

A

You can see the sample spiffy certificate, how there has issued by Athens, and you can take the principle from that, particularly one eye. The next step towards is, like hey I, have said. The principle is authenticated request whether he's allowed to do that operation. Its next step is authorization. We want to natively use the enjoy our back. Ok, so that's in line with the request, and it's not that we are making a request. Our additional network of a local call to authorize the request.

A

We want to make sure it's in line right and what happens when you have an inline. Our back, like NYK, request, take the principal validate against this policy and he responds back is allowed, are denied. So now that and boy has to make the decision.

A

That means that anyway, I need that information about how the policy information- how do I make this a no motive right, so in each team, I mean the our back is defined as integrated, Satan's, well tooth custom resources get created when it's called service, road and other ones called service road binding, and when these two resources were created, the pilot responsibility is to take this to custom resource and then push the configuration to NY. So the NY can't make the dish and locally right in part of the request role, but I was mentioning.

A

Athens is our our back system. Okay, we want to define the are back in Athens. We are not going to create this year, CR and service role and service or binding, but everything which is manually are to a pipeline. So we build this controller. What's this primary job of this controller is to let the user define their policies in Athens, we take the information from Athens and create this to resource, which is custom role and Istanbul binding, and once you create that one, then pilots take the responsibility to push this configure in.

A

Why so NY can make edition very locally? So what happens over here? You are a service provider. Okay, what you're, trying to say is like hey, I am going a low get for me. I'm, an integration or being integration service provider. I am going to allow get operation for all my v1 user API star every API I'm going to allow to this particular role.

A

You can see there is a role creator called Y finest customer care, prod access and non products us so whoever is member of you can look at the role definition partner to particular guy. You can see in there to your left. You can see there your role definition, which says that he way micro-service custom care products s. So whoever is part of that particular role is its authorized to make a get call are allowed to do the get call on this set up in EPS. So we take this information.

A

What you're trying to do is, if you want to allow service a if you also want to allow service, be. All you have to do is go to the Athens UI and add the service as part of your member list. If you're done in next few milliseconds, we take that information and push it to see. We create the to see us and then n more pilot takes care from there and pushing to NY, so the authorization happens very very quickly. We were able to push this computation very dynamically.

A

This is high-level how the the controller works. It takes information from Athens and then just creates the tools here and once that, yesterday, its pilot responsibilities push the configuration so that he can make the Dacian very locally. So, with this approach, we can see that the yard in kubernetes and we're running heavily on communities, and we got easier to do the authentication authorization and how we have integrated with our own CA and our own identity provider, Athens.

A

Right now that we have done this work and in 2020 we going to make sure that all other services also running mutual TLS and encrypted at any gate and at rest. How did we get here right? We have like thousand plus application on board are out here in last two years right. That means that we have to give a very good user experience who wants to unroll to the service to gain all the benefit out of it.

A

How does our developer experience looks like who went to use our kubernetes cluster and enable bring the workload to communities and then enable mutual TLS and things? Let's let more on my talk about the developer user experience.

B

So when we embarked on our journey with kubernetes, almost three years ago, we were faced with the enormous challenge of having to migrate hundreds of our existing applications onto this new platform. As today's mentioned previously, all of these applications were had their own diverse technology stacks and they had their own independent deployment pipelines. So, right from the start, we wanted to ensure that we present a simple and concise developer experience for our developers, so some design goals that we kept in mind. The first one was simplicity.

B

We wanted to design an easy and self-service onboarding procedure so that our developers would not need a lot of hand-holding from our side when they were onboarding their applications onto this platform. The next thing that we realized was that when you deploy an application onto kubernetes, you need to know you need to have the knowledge of multiple kubernetes resources, for example, is like a deployment service, ingress, etc. So this would prove to be a very steep learning curve for our developers if they had to go and learn all of those things.

B

So we decided to build an abstraction layer which would take away some of that in-depth knowledge that was required. So we built a single, simple yamin that would be able to configure all of their application configuration, and this would make it an easy and fast migration path for these developers. The next mole that we kept in mind was flexibility. So while we took care to define certain same defaults, we want also wanted our developers to be able to override those with application, specific settings and the last design goal that we kept in mind was stability.

B

All these tools and workflows that we were building, they were going to be something that developers interact with on a day to day basis. So in case we pushed out a bad change. It had the potential of impacting hundreds of our developers and it would lead to a lot of sort of frustration with our system. So what we did right from the start was to develop a hook to hook up a comprehensive acceptance test so that every change that we're pushing out was thoroughly tested before it actually reached our consumers.

B

So the simple Amal interface that I mentioned that had two main characteristics. The first was that the settings should be self-explanatory so that, just with some simple documentation, developers would be able to understand how they need to configure their application. So one example is there's a routing section, so it's very easy for them to understand that. Okay, all the routing rules for my application need to be defined over there, and the second thing that we kept in mind was that they should not need to.

B

They do not need to know anything about kubernetes resources specifically, so while we could define their routing rules in the routing section, they didn't have to know anything about a kubernetes ingress resource or an sto virtual service.

B

So in order to enable this, we built a standardized list of templates for every kubernetes and sto Yaman, that audio source that we want to create on our cluster. Along with that, we built a proprietary templating engine. What this engine would do is it would take the set of templates take on the default settings that we've provided and also take the user specified overrides, combine all of them and create an expanded list of amell's.

B

So this diagram over here this. What is in black that kind of shows the artifact that we create the amell's that we create are created on a per application environment and data center basis. So, for example, if there's you want to create amell's for your production environment, and you want to deploy that and to say a US East one Colo. We actually create a specific set of amell's designed for that. Another thing that you can see here is that some of the application provided settings are there in the expanded ya Mel.

B

So you can see that the app name that they specified, which is appended with the data center, that they want to deploy to as well as they told us that they want to add as a logging sidecar. So we made sure that we add a Splunk container along with that application, so that their pod has a sprung sidecar running with it.

B

Apart from these standardized kubernetes or templates that we generated, we also wanted to develop certain build templates. These build templates are based off screwdriver, which is Yahoo's CI, C D flat build platform. This has also been open sourced, so do please go check it out. The reason for building this standard build template was that we wanted to enforce certain steps in every applications. Cic D flow, so the steps are pretty generic, so you would want to build your code. You'd want to run unit tests.

B

You then want to build a docker image out of it, and then we publish it to our internal docker registry. This would be followed by the ml generation that I discussed previously and then. Finally, we would want to deploy this entire artifact to a target, kubernetes cluster and then we'd run up the applications functional tests against it. So this would guarantee to a developer that whatever they've built locally is working as expected, even when it's deployed in kubernetes.

B

Apart from these specific steps that we've listed here, developers can also inject their custom steps so, for example, if they want to run sonar cube as a part of their pipeline, that's something that they can very easily just inject as one of their build steps.

B

So now, I just want to go into sort of like a deep dive where I'll take an example scenario. Just to illustrate how simple it is for a developer Yahoo to configure a fairly complex use case. So if I'm, a developer and I want to deploy C or node.js 10 application. The first thing that I will do in my abstracted Yambol is I'll, specify a base image setting which will say that I want to deploy an app which needs node.js 10.

B

The next thing I want to do is I want to configure the routing rules for my application. So, for example, I want to make sure two outs through sto, ingress and I want to make sure that it routes on port, 443 and I also want to add a custom alias. So all of those settings can be put under the routing section that is defined there. Another thing that I want to add for my application is that I want to add in sto, sidecar and I want sto to enforce two things.

B

For me, one is to add strict mutual TLS and the other is to make sure that every outgoing call from my application is restricted to a whitelisted set of endpoints. So all of that configuration for my sto sidecar and would be, as you can see in the sidecars sto section here. I can also specify certain like resources that I need for my Envoy sidecar, like memory or CPU.

B

So as you can see that all those settings that I mentioned, which would have resulted in multiple kubernetes or sto amell's, can be abstracted, and this describe very simply in this one yeah Mel that you can see here on screen. Of course, behind the scenes we do go ahead and create all the animals that are required. So you see your kubernetes deployment you'll see your is to your policy virtual service gateway.

B

Everything that you need to make that configuration or reality is actually being created behind the scenes, but developers didn't need to know any of it, so they can reap the benefits of all these features without getting too deep into kubernetes or sto.

B

Apart from all the features acceleration I have mentioned so far, there's a lot of other cool stuff that we have on our cluster. That developers can use, for example, rate limiting fault injection, distributed tracing traffic shifting, as well as securing the egress for your application.

B

Just one more thing that I want to call out is all of the tools that Suresh mentioned previously. All are Athens integrations like, for example, the at insist. You are back controller, they've, all been open sourced and aside from the ones that we've mentioned, we also have a couple more in the field of policy and security, for example, security, benchmarking.

B

So all of these have been open sourced and if any of you is like a kubernetes cluster operator, they are fairly generic use cases so definitely do go check them out and if you have any features or any issues that you see with it, we're very open to any contributions.

B

That's it for me, I'll pass it to Suresh. To conclude thanks.

A

For only I'm going to conclude right, we're running out of time right, so we got this kubernetes and it's just act today and we are running over right. It helped us to modernize and standardize. Our stack and very importantly, other aspect is like it: helped our product velocity and improve our product developer productivity. Finally, it also improved our application security. It's enhanced it and one.

A

Another thing else I want to call out is like our customer relations grown up because we've been packing different set of workload into the same cluster, so the equation is very, very, very high, but all the benefit which we have out there and all the critical applications, such as Yahoo, Sports, Yahoo, Finance and Yahoo homepage, is great millions of RPS. It's completely sort of cognitive stack, which we have very very, very proud of and again below kubernetes.

A

Thank you and we are hiring and if you want to join me in the journey of pushing towards contamination and then bringing other set of application out here and the journey of castillo kubernetes OPA, please join me and you can reach out to me in this mail ad and if you have more questions, we have a booth at eight. Please come over. There I'll be happy to answer any questions and any hiring questions. Thank you.

A