Cloud Native Computing Foundation Kubernetes Forum Bengaluru 2020, 27 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sponsored Keynote: Multi What???? Multi-cloud, Multi-cluster, Multi-tenanted K8s - Gaurav Mehta

Description

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects - Learn more at https://kubecon.io

Sponsored Keynote: Multi What???? Multi-cloud, Multi-cluster, Multi-tenanted K8s - Gaurav Mehta, Field Engineer, Rancher

Gaurav Mehta will share some general Kubernetes trends and discuss some personal experiences after being involved in a multi-year transformation at a major financial services organization. Gaurav will highlight the challenges and the benefits the organization is now reaping from this work, all thanks to the power and flexibility of Kubernetes.

https://sched.co/YW4E

A

Morning, Bengaluru and Gaurav unfilled engineer based in Australia, so I'm just going to talk about some of my experiences with kubernetes the general trends we are seeing so first up, you know this was a survey that you know we conducted we sort of notice that nearly every organization is running Multi cluster kubernetes, ninety-one percent of the teams are running multiple kubernetes clusters, 71% run kubernetes on-premises, which is the private cloud. Sixty seven percent, you know also running in public clouds, and you know what we're seeing right now is. 15 percent of of you know.

A

Customers we are talking to are investigating running kubernetes on the edge I was fortunate enough to be sort of involved in a massive kubernetes transformation. At a large you know financial services firm in the inside region. We sort of started small and ended up running. You know tens of thousands of containers across you know. You know multiple kubernetes clusters and you know we supported 300 plus developers, and you know it was a multi-year journey like I said, but it was managed by a team of five. So you can imagine the challenges we faced.

A

The level of automation we had to aim for because we always wanted to keep ourselves as a lean team. So what I'm gonna present now or talk about is you know, like the blooper reel from sort of what we learned over those four five years?

A

The first stop was multi-cloud I'm, not sure. If anyone follows Cory Quinn on Twitter he's a cloud economists and he's hilarious I would recommend you definitely you know, have you know see what he's up to and you know look at his sarcasm. It's it's pretty hilarious. So this was our challenge when we started right that everything worked in theory, but nothing worked in prod because we had lots of dev environments in a public cloud and then we had a parts to production which was hosted in a private cloud which meant you know.

A

Everything worked in theory, but as soon as record to prod you know nothing worked and all of this automation was written using puppet. We had different puppet modules to build in the public cloud and different puppet modules to build on prim. So we started investigating containers. You know about five years ago, because containers were good at abstracting, the under like cloud provider. You know packaging all the dependencies, and once we had containers done, we needed an orchestration mechanism to run containers at scale.

A

So that's where you know we ended up with eventually you know we started with cattle and ended up with kubernetes as the container orchestration engine. The benefit of you know of this engineering work for us was that we were able to build immutable and consistent infrastructure pipelines. So if someone wanted a kubernetes cluster, it was as simple as clicking a button in a in our CI, CD, tooling and you'd have a kubernetes cluster with all the base services installed and deployed within minutes. You know, and we were always looking to speed up the process.

A

I remember the first time we sort of built a cluster. It took us about three hours to build the cluster, deploy all the workloads and have all the basic services up and running. We sort of kept chipping away at it kept iterating and eventually we ended up. You know being able to build production grid clusters with all the workloads deployed in less than an hour, and you know because we got pretty good at it.

A

We were able to solve the next problem, so I'm a big star, Wars fan, and in my personal opinion this was the biggest you know flaw in the plotline in in the original. You know trilogy that you know I mean I would thing the Emperor failed his BCP plan, because it didn't have a second dead star. So the idea was that you know we wanted to do sort of multi cluster and the reason was that we would lean team. We didn't want to work at night. We were all lazy.

A

We're, like you, know we're not going to stay up at night to rebuild our clusters. We did immutable infrastructure anyway, so to patch it it meant we were going to delete a cluster and build it back, and you know the benefit of that was, you could easily sort of you know. Do security code updates during business hours. We didn't have to ask for outages. We basically told business. This is the time we are rebuilding one production site.

A

Nothing's gonna happen you just you just need to be aware that something's happening and that definite helped us write the fact that we could rebuild production clusters in an AR. You know- and you know it also said to some human error. I'll talk about a personal story, I'm, not sure if anyone's done this themselves, but I managed to delete a production cluster accidentally, because I was trying to delete something else, but I didn't look. I was trying to multitask too much.

A

I was like I need to delete this cluster, but I didn't realize the drop-down said: fraud and I, clicked, ok and then I watched in horror, oh I deleted fraud, and it was too late because I could I just had to wait for it to finish its job in rebuild, but fortunately no one noticed anything. I just notified business that you know we lost a product cluster in an hour. It will be back.

A

You shouldn't see any you know impact, and you know it was amazing, because you know people suddenly saw the value of all the work. We did right even avoiding human error, so all of this infrastructure that we've built initially was for, let's say one dev team soon would spread out to other dev teams that you know they're running containers, they're running containers at scale. Their deployment times went down from three, so our code releases used to take two to three hours.

A

It became came down to five minutes, so everyone wanted to you know, use our platform that we've built like Tim's mentioned. Kubernetes is the building block and we sort of built our platform on top of K 8 and which brought us to the next challenge was multi-tenancy, and this is my favorite example. Reichian um I've been on call when I after I built the platform for literally three years, so I've seen the good the bad the ugly. The first thing was this 3:00 a.m. call I'm, not sure.

A

If anyone's had a late-night call asking something's broken, can you fix it? So the first 3 a.m. called was my pods failed. You know, I am running two parts. One of them's failed, so I'm, seeing 50% failures, I'm like yeah dude, just delete the part. This is like 3:00 a.m. it's like no, no. No.

A

This is fraud, I'm not going to delete the anything you have to do it for me, I'm, like dude, you can't make it worse, just delete it and the scheduler will spit out the new one and both of us can move on with the life, but he was too scared, so at 3 a.m. I got and deleted the part and I said dude happy. So the next you know day I came to work. I did a lot of rage-fueled coding, so I said you know, lots of devs are consuming a platform, let's enforce some standards.

A

So, let's see you know, we wrote our own CLI to sort of you know, wrap around cube cuttle at that point to allow people access to our cluster selves like we are already parsing the manifests, so I can just apply some best practices to make sure there are, let's say to start off with readiness probes on your pods. You know liveness probes. So if you do not have one of these, you know your boot load cannot be deployed to any environment.

A

Then I went on to add, you know container image scanning memory limits, so we could, you know size that cluster accordingly and you know still we had issues. The next issue was our back again. You know another midnight call a dev calls me up. He was trying to release some code. You know a test environment I, don't know why he was doing it at midnight, but he called me to complain that you know I I was supposed to delete the part, but I deleted the node and I was like that's ok, which environment is this?

A

Is this prod this like? No? No, this is tests I'm, like that's fine. You know this is Saturday night I was like Monday I'll. Look at it he's like can't you look at it right now, I'm like dude, it's test. Let me sleep we'll do it on Monday, so Sunday afternoon you know, I was sitting home, not doing anything important. So let me just see you know what has he done so that cluster was supposed to have 20 nodes and 10 nodes were missing, so I called him up at what point?

A

Do you realize that you're not doing the right thing right? You were supposed to delete a part, and now 10 nodes are missing. Like you know, doesn't it make sense to you that you're something unintended is happening here, so that led me to enforce our back at the cluster level that you know you shouldn't be doing something you know, even if it's dev that you're not allowed to do.

A

The next challenge, I heard was like container scanning, so I'm, not sure about the devs here, but our devs only wanted to check in their line of code, deploy it and move on. They didn't care about what happened with it or you know if there were wonder abilities in the packaging, so I enforced container scanning to force people. But you know you need to do the right thing and sure enough.

A

The bills started breaking because they were Wonder bilities in the you know, container images- and you know the next request- I get is in your CLI. Can you write a flag to bypass container scanning, so I wrote an extra flag in the CLI to bypass container scanning sure enough. Every Devon of those 300 days is now skipping.

A

The container scanning and I'm like what's the point of this, but me being me and I'm like a sneaky guy, so I went in added some Prometheus metrics in the CLI, so we would push all the metrics through Prometheus gateway, so I knew. Who exactly is you know using the flag and which service has how many one durability so I could go to you know their managers and say: do you want to do something about it sure enough?

A

They were like yeah, let's enforce container scanning at deployment time, so I enforced deployment scanning at deployment time, but because everyone was skipping the built time scanning, so it caught them out at deployment and I got the next feature request. Can you skip deployment scanning in dev tests just scan it in stage and throttle I was, like sure, I mean? If that's what you want, you know you are. My customer I'll write that extra add-on to it. It's a small flag for me.

A

So I wrote that next you know flag that they will just skip it in dev test and fraud and stage we will enforce the standard and that led to the next ya know midnight call we had a prod release, but we can't deploy to prod, because you know your container scanning is not allowing us to deploy it to fraud. I was like okay. Did you paste it in dev or test yeah dev test winner? So when I asked them, did you test it in stage actually and they were like?

A

Oh, we went from dev test to prod, so we didn't really pick up these. The image vulnerabilities in stage I'm, like you know, that's what it's there for so they're like. How can you disable it right now? I'm, like no I'm, not disabling at night right now, you go to business and explain to them why you can't deploy to prod right now and sure enough. That was the best way to sort of educate the developers.

A

You know once you once they realize the codes not going to go to prayer prod next day there was a CI plan to run image scanning every few hours on the base, images and alert them to tell them that now there's a vulnerability and it will break the production deployment, so you better patch it now so after I was done in about 6, you know applying incremental standard enforcement. This is what we ended up with.

A

But the benefits of this were amazing. Right, for example, we had this improved security, posture, I, remember, Spectre, meltdown bugs came out. Most of the organization was talking about how we gonna patch stuff and they came to us. You run your own infrastructure. What's your plan and I was like? Oh it's already in in Prada, it's been in. All our clusters were rebuild everything's running in prod for a week. What are you talking about and they're like really I'm like sure?

A

Next? Was the speed to market right a lot of the times we could sort of you know deploy. You know code which hadn't been through the full integration cycle, because it was a business demand to prod, simply because of the fact that it was as easy to roll back as it was to deploy a new version and the last but not least, was ability to respond to events.

A

I live in Australia and we've had like really wild weather in the last few years, so I remember, Christmas, Eve I got to get a call that there is a massive thunderstorm approaching and the weather department's issued an advisory. So we expect you know, the general trend is, after a weather event, everyone logs into the online portal to check whether they are covered or not, and if they are covered. The next thing is a claim to sort of you know, claim for hail damage and sure enough. That was the business call. Can you increase capacity?

A

So five minutes later we had scaled out our product production clusters, all of them by a hundred percent capacity, and you know that's what you know: values to the customer that they were able to validate the policies. You know launch their claims and you know, consume the service as it was designed to, and that was like the first real use case where I found that you know this is what matters to the end customer and to the users.

A