Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tutorial: Debug Your Kubernetes Apps - Arun Gupta & Re Alvarez Parmar, Amazon

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Tutorial: Debug Your Kubernetes Apps - Arun Gupta & Re Alvarez Parmar, Amazon (Limited Available Seating; First-Come, First-Served Basis)

Please bring your laptop fully charged as we will have limited charging stations available in the room. Your Kubernetes application is running well, and then all of a sudden the service stops responding. How do you debug? You created a deployment but its not coming up. Is your pod status shown as pending? How do you debug deployments and pods, get their logs, see the filesystem layout? Horizontal Pod Autoscaler is not scaling pods. Is your cluster running out of capacity? Or are the metrics not available? Having DNS lookup failures for services? Is your PVC status shown pending? Is kubectl not able to find nodes? This session will be loaded with different ways your applications on k8s crash and burn, and more importantly to recover from them.

https://sched.co/UaeY

A

All right good morning, good morning, thank you very much. My name is Arun Gupta and today I'm here with my colleague ray.

B

Alvarez and I work with AWS and the specialist container solutions- architect, fantastic.

A

I'm a principal open source, technologist I'm, also the CNC F board representative for Amazon. So today we will talk about debug, your kubernetes applications, just like anybody. We have a twitter handle and we both work for AWS. I know this is cube con. But now, when we talk to customers about continued rising, modernizing their application, we always ask them why they want to use kubernetes most of the time. Kubernetes ends up being the answer, but we want to emphasize the fact that kubernetes is not a golden hammer.

A

It may not be the solution of every problem that you're looking. We hope with us, but it's not the case so make sure you are looking to solve the pain points, the problems that your customers are facing and then applying kubernetes to solve those problems.

A

Another principle that I really found recently I really connect with that, the more you think you know about kubernetes, the less you think you know you know: I mean every time you know you start peeling. The onion and you know it's more tears coming out of your eyes. So important part is to understand. You know we'll see that in the presentation, this presentation is all about the different ways your apps can fail, and how do you recover from that?

A

Essentially, so, in this talk, what we intend to cover is several aspects here we put asterisks on the last two topics, because we think just like kubernetes we are oversubscribed. Hopefully we can cover all the topics here, so starting with cluster design. What are the basics? What are the errors that we are seeing with our customers over there? Networking cube kernel, CLI pods, so we'll talk about every places.

A

You know where there are errors happening in these areas and how we can help debug those load-balancing, ingress, monitoring and then time permitted, we'll look at resource reservation and stateful sets as well.

A

So, let's get started by cluster design, you know before we even jump there. I want to get the basic concepts out there that when we are looking at and how are they designing their cluster, what are the basic principles that they need to think about?

A

One thing, I do remember, is go down here, alright, so let's talk about the core kubernetes components. At a very high level, if you think about a cluster set up, you know you want to create a highly available cluster. So add in Amazon language. We have this concept of a region which is a physical location around the world where we cluster data, centers and in a region we have multiple availability zones. These availability zones are far separated from each other for H a and disaster recovery.

A

So what you're, showing here is how do you set up a kubernetes cluster in a region spread across multiple availability zones? So in terms of kubernetes components, what you have is really of controller and there are a bunch of controllers over there, which will dig a little bit into deeper into it. Then HCD is where your state of the cluster is persistent down there, and then you have worker nodes, which is where basically, all of your containers are running.

A

Essentially so in terms of setting up a cluster, you want to make sure that your controllers, HCD and workers, are spread across multiple availability zones. That gives you the basic principle of H a, not a single point of failure. You know yes, region is basically where you are deploying it, but there are strategies around that, but essentially what you're doing is you're spreading your workload across multiple availability zones in case one availability zone goes down for whatever reason geographical disaster or whatever it is okay. So that's the key component that you want understand here.

A

So how do you set up a kubernetes cluster first step is to really create your controller and when we are talking about controller, we're really talking about scheduler controller deployments, that controller replica set controller. All of those controllers that need to run together together, let's call them as master. Basically, so you create a controller node that is typically called as control plane. Then you need to setup HCD, and the decision that you need to make over there is will HCD be co-located with your master, or will it be running on a separate node at Amazon?

A

We prefer that it runs on a separate node because again you know it allows you to reduce your blast radius and allows you to upgrade HCD by itself as well, and also the needs for your controller, which is more CPU bound as opposed to HDD, which is more disk bound. You can set up the right see two instance types accordingly, then you have cube Caudill CLI, of course, by the way which you connect to your kubernetes cluster or to the control plane.

A

So that is your control plane essentially, and the only way you can talk to your cluster is using cue, cuddle and directly talking to the control plane. Then you install your worker nodes and the worker nodes is basically you know all of your ec2 instances where your containers, pods deployments, etcetera, all cannot run.

A

So you have your control plane, then you have a data plane and you attach that data plane to the control plane, and then you deploy your apps your add-ons, whatever functionality you want on top of that kubernetes cluster, using cube, Carlo, very simple process, but devil is in the details and that's where things start breaking apart. Hopefully you are making profit out of that application and go back to the first slide. Remember what is the pain point? What is the application that you're trying to deploy over there? How are you trying to solve that problem?

A

It should be in your mind. This is just a process, as it was saying in the morning is not the destination. It is a car to the destination, so remember the destination in the mind when you're designing your kubernetes cluster. There are a lot of considerations that needs to go into in your mind: should I use control, plane, self-manage or should I manage the cluster by myself when I'm designing the controller nodes, what kind of ec2 instance type and I'm just using AWS terminology, but these are applicable to across the clouds?

A

What kind of ec2 instance types should I use? What, if my number of pods on my worker nodes increase? How should I scale my data plane? How should I size my lcd securing my HDD backing up my HDD setting up a raft consensus protocol day? One is okay. Now I've got a hello world application up and running what about my day, two upgrades patches, CBE's who's, going to do all. That is that my core competency always think about that part of it or you have that capability or is this morale booster?

A

That's something goes on you resuming hey, I'm, a kubernetes sre, so think about what is the motivation over there? It's okay to have that hundreds, even by the way, but some of the design considerations and again these are very design considerations, but you need to look at when you are designing your own kubernetes cluster. That's where you know we created Amazon eks Amazon eks is Amazon's version of a managed kubernetes service, and essentially what we give you is a managed control plane and in that control plane you don't need to worry about it.

A

What is the ec2 instance type? How often HDD is backed up? How is it going to be a che? All of those are given to you by default and on Monday actually of this week we announced something called as managed nodes, but what that means is the full cluster if you want it, including the control plane and the data plane is fully managed for you, so that capability now exists in Amazon II cares.

A

But if you want to bring your own custom worker nodes, you can totally do that because that's how we initially launched essentially ok so think in terms of when you're designing your kubernetes cluster or is it a che? Is it we, our security back hum what kind of instance type upgrades and all that capabilities?

A

Now you look up. Look at the concept of cute cuddle as we talked about it, so your cluster is created. So you take your Q cuddle and that's how you talk to your cluster and then the cluster is really what is deploying or the API server, essentially, which is listening to all those requests from Q cuddle, understands those requests and then deploy your objects into the data plane over here. Ok, so that's to coddle is basically one CLI that manages your equipment at is cluster.

A

Let's look a little bit more into the kubernetes cluster design so that we understand which what are the things that can potentially break and soon we will start getting into the real deal over there. But if you think about the master, node components, the first one is an API server which is sort of because, if think about, kubernetes is nothing but a rest endpoint. It listens to REST API that is being sent from the coop cube Cardle. Essentially, so there is an API server that needs to be sit there.

A

Then there is a controller manager, as we talked about earlier deployments, that controller replicas controller replica, set controller persistence, set controller, you name it all of these different controllers are existing and at a given point of time and the request is received, they make sure that the desired and the actual state are reconciled. Essentially, so that's what they take care of. Then there is scheduler, which is also a controller. So after all, the status persisted into HCD the scheduler kicks in and the scheduler says.

A

Okay here is a part that has not been assigned to a node. Let me rank all the worker nodes that are available based upon my predefined algorithm or if you are giving me a custom, algorithm I'm gonna pick a worker, node I'm gonna assign a part to that and again persisted into HCD as well, and then, of course, there is a sed which is sort of the back end on where the entire state of the cluster is persisted.

A

So those are sort of the core master node components over here, and things can really break at all different levels over there in terms of HDD design. You know it follows the raft consensus protocol. So once again, the recommendation over there is to have at least three @cd servers spread across multiple availability zones. So typically you know when the request comes in to the QB API server, it writes to the one HCD server, which is in the same availability zone, and then it replicates to other it's eventually consistent.

A

On the worker node component site, what we are looking at is there is cubelet that is sitting on each node, think of it as a node agent, essentially, and that really handles the communication between the API server and the worker node, and that is the one that is really running your node and communicating with the API server in terms of the status and the what needs to be done and how many replicas etc. Well, the replicas are designed by API server, but cubelet is responsible for the lifecycle of the container on the worker node.

A

Then there is queue proxy, which is basically handles the communication between parts nodes and the outside world, and last but not the least, there is cree or container runtime interface or cry, which basically says. Okay, there are multiple ways by which the containers can be run. I could use, docker, I could use container ID I can use cryo, whatever technology Sokka. Now it's a very kubernetes centric interface, but gives you the independence from what the underlying runtime technology could be.

A

Now, in terms of creating a cluster, the way we look at you know creating an Amazon eks cluster is very straightforward. Eks cuttle is the official CLI which is created by one of our partners. We've works, so essentially, what you say is eks kernel create cluster. We were looking at the keynote this morning talking about convention over configuration, and this is exactly what it follows. The moment you say ich es caudal create cluster. It just has a whole bunch of defaults. What region, how many instance types, how many worker what instance type?

A

What is the army all that default, and then it gives you a default eks cluster? If you want to configure the cluster, if you want to create a cluster by your specific design, I want to change. You know the number of nodes, the instance type or the SSH keys, or what kind of logging property properties it has sure you can specify them on the CLI, but you can also specify them in a configuration file.

A

So, following the practices of how you create your ETS cluster, persisted into the database persisted into the github repo, so essentially what you can do is you can create like a configuration file and in the configuration file. For example, in this case, I am saying my cluster name is debug gates and I was in the region, US East, one then I'm specifying a node group, and the node group has m5 extra-large instance type and I'm, creating a four node cluster I'm, also enabling SSH and I'm saying use my US East one key.

A

So essentially, that's that's all and in the last fragment, is where I'm saying enable these kind of logging. So my control plane logging is already available. That is the most useful T useful tool. Typically, when you get to when things go wrong. So it's very important to enable those kind of logging over there.

B

Thanks Erin, so let's look at networking and when Erin reaches out to me and said we need to do this presentation and we need to find out all the ways communities clusters have failed on AWS a result. Two or three friends inside Amazon and I started asking them questions like what do you think are the number one issues that customers run into and unanimously everyone came to me with the same response that the number one problem on kubernetes is generally related to network.

B

So there are two core components that were we are looking at when it comes to networking problems. The first one is the C&I. Now in kubernetes, kübra not provide its own networking stack, you can bring in your own network provider. Some have some functionalities like network policies on eks. We are using the Amazon VPC CNI, which gives you the ability to create to give your Pazza VPC IP address, and the next thing was core DNS.

B

A lot of work has been implemented in core DNS in the last few months in the fad last few releases, since it became standard in in 1.11.

B

So a lot of improvement has already happened, but when you're running clusters that include thousands of pause or hundreds of nodes, it does not scale automatically, and there are certain steps that you need to do so we're going to take a look at some of those use cases today.

B

So let's look at the C&I. What is Sienna Sienna is a container networking interface and on eks we use Amazon, V, PCC and I've a default, and that gives you the ability to give your pods and your worker nodes the same IP address from a subnet within your V PC, and if you've ever used cops or any other thing you, you might be familiar with things like calico or flannel or Canal. These are overlay and networking technologies that include a little bit of overhead, the compute overhead to run on the worker nodes.

B

But also now you have to deal with another layer of IP addressing that you don't have to deal with in Amazon, V, PC and I, because what the IPS that you're seeing for pods and your worker nodes are real VPC IP address, you can ping them. You can connect directly to them and you can do things like BBC flow logs, etc.

B

So if you look at how the cni works in in eks c- and I basically uses elastic network interface, which is a network interface that you attach- or it's automatically attached to an ec2 instance and that's basically providing the networking capabilities of each ec2 instance and that's how you connect your network, so each ec2 instance gets an en I attached to it, and Amazon BBC CNI runs on top of en I. What that means is it uses en is to give IP addresses to pods and ec2 instance.

B

One of the IP address that you have in en I is reserved for communication, so typically use the max number. You see deduct one, because that's what you're, using for communication back and forth three VC pods as I said, receive an IP address directly from the subnet. So that means that you now have a limitation of the total number of paths that you can have in your cluster. First limitation is the size of the subnet.

B

So if you have only hundred IPs in the in that subnet guess what you can only create hundred pods anything that any pod that's created after that is going to remain in pending state unless you have, unless you add more IP, address or increase the instance size. So you should really pay attention and plan for growth before you're, starting your cluster. Make sure that you get a idea about what is going to be that upper limit for your pods and then create a network accordingly.

B

So, let's see we create a deployment. This is a very highly sophisticated application that I've written and it's very highly sophisticated output. It says hello world to the world I'm very proud of it and I want to run it so I basically go to go to kubernetes and say coop Ketel create my deployment, which is a hello world application, but of course I since this is such a high transaction very, very important application I have to make sure that it runs at scale. So what I did was I said scale my replicator 240.

B

Let's see how is the performance of print hello world at 240? Replicas, so I was very proud that it's gonna work and then this happened, I want to say get deployments and I was very happy that two hundred twenty two of them were adopted, but the remaining were orphans or pending states. So I got very curious, I'm one of those users that learns by actually doing so. When the idea comes to my head, I directly go into doing without skipping the very, very low importance task of reading documentation.

B

So I, you know, kubernetes works and I just will go ahead and scale the two millions on unlimited scaling and I who I should have that, but it didn't happen. I only have 222 pots. What went wrong so I try to get get pods and seems like some of the pods may be in pending state. So here's another cool thing. If you don't know about field selectors, these are really important, especially when your cluster becomes larger and you may be running hundreds or even thousands of pods fields.

B

Lectures really come into play and you can filter them so over here, I'm. Just doing a give me all the pods that are pending and I see a bunch of them are pending, so I want to troubleshoot a little bit more now this time I can use, get events cube. Caudill get events is going to give you all the events in the cluster, but over here, I'm, not interested in anything. That may be information only so I'm, only looking at warning and if I dig a little bit deeper into it.

B

I'm shown something that looks like this and I see that there's an interesting part. That's the part I'm interested in my hello, world application and I see that it says fail to assign IP address to container. So that's the number one issue people run into because guess what I did not read?

B

Documentation and kubernetes was supposed to solve world hunger, and yet here it is, it can't even create my 240 pods, so I want to take a look into it and I then started reading documentation, because that's what you're supposed to do you work first fill and then read documentation. So I looked at this nifty component of the CNI and it's called ipam d and it basically does two things. First of all, it maintains a warm pool of IP address from the VPC.

B

So let's say we'll cache of 10-15-20 IP addresses and as pods are created, is going to assign an IP address to each one of them and as pods get destroyed, is going to then take that IP address and keep it at this warm pool. So, as you mentioned, EA and I basically uses one IP address for communication, so the max number of I max number of IPs I can get in in an ec2 instance.

B

There's a simple formula for that: I, don't know if you guys can read it so the formula really is you take the the number of en eyes times that by the IP addresses that each in that eni supports minus 1, that's the one it uses to communicate back and the number of IP addresses you can get on that that ec2 instance is going to be a min function. So, whichever is smaller, that's going to be a limiting factor.

B

So in my cluster here, I have an m5 x-large ec2 instance type and it can support up to four yen eyes and Ichi and I supports 15 IP addresses yes,.

C

B

Increase the font that could be a challenge. We've never done that. I, don't know what else is going to break yeah.

B

This is better than hello world, so I have an m5 x. Large cluster for en is 15 IP address, I'm really bad at maths. Don't let me do this year. Yes, 456 was the number I was trying to get to, so it can support up to 56 IP address where the four times fifteen you get it. So that's a 56 is the is a maximum en I that it can support the default cluster that I created in that default.

B

Subnet it supports up to 8,000 192 IP address it's a slash, 19 I knew it didn't need to read it because that's how I calculate ciders, so what the maximum number of pods that I can run on this ec2 instance is going to be directly related to the maximum or IPS that I get on this ec2 instance. So over here and my limit is going to be 56 and 56 is a maximum or a pause that I can run inside this ec2 instance.

B

One recommendation at 116 is that you should not run any more than 100 pods per node and if you want to know why click on that link, it should tell you exactly why. So in this situation, I have 4 ec2 instances for m5x larges. So that means that 56 times 4 is going to be 240 is.

C

It going to be 224.

B

224 and that's why my pods stopped at 200, why did they stop at 220, because I may be running other pods that are also consuming IP address? So that's the number one thing to keep in mind is that you always want to make sure that you are planning ahead. You have enough IP addresses, because if you don't, your application scaling is at one point going to come to a halt and your cluster will break your apps one skill.

B

So, in order to do that, we've created the cni metrics helper, open source tool, and it is a very easy tool that you can deploy inside your cluster and it can report back metrics like what does he know? Yp address is available or are there any errors in my cluster? So let's take a look at deploying that and very simple. First of all, I'm going to create a policy. This iron policy is basically going to allow. Whoever assumes this role to write to clog watch directly. So I will write this policy.

B

The next thing I'm going to do is attach that to an ec2 instance, which is my m5 Excel in this situation, you can, alternatively, also use what we release recently. I am mapping to a service account, and you could do that too. The next step, I'm going to do, is I'm going to apply this yamo file, which is going to create a bunch of resources inside your cluster.

B

But the most interesting thing is that it creates a pod inside your coop system namespace, which is CNI metrics helper, and this mattress helper is collecting information and then providing that to you on cloud watch so once I deploy. This I can go to my cloud. Watch and I can really quickly look at how many IP addresses I have available and and how many IPAs are in use. I can create some some really nice alerts to send me a slack message in the middle of night that you're running out of IP addresses all right.

B

So let's look at core DNS now this was a number to think that that people told me customers run into so. Let's look at core DNS core Deanna's use was before core DNS came in, we had cube DNS core DNS was g8 in 1.11. So if you have clusters that you created there are 1.11 and you long forgot about it and they you should switch to core DNS. But if you create a cluster after a one-goal lemon by default, you're getting core DNS and succour. Dns is highly scalable.

B

Highly customizable DNS is a provider for kubernetes and it uses a core file to control. It's basically the configuration for fork or DNS, and the core file is stored in a config map like all good apps do and it's stored an accordion s, config Mac, which you can directly manipulate. So if my my, if I'm facing some problems with DNS resolution in my cluster, the first thing I will do is go. Look at your pods.

B

Go look at your DNS pods, so in eks, by default, we'll run two replicas of core DNS deployment in your situation. Maybe one maybe three. We recommend that you run at least two at a time if you're positive, running and you're still not able to get named resolution, or you have latency. The next thing I would do is check your service. So there's a core DNS service that you can look into and make sure that it's up and running.

B

If that doesn't work, you can then add logging to your core DNS and here what I'm going to just edit my core DNS conflict map and I've added this.

C

You guys see this this, yes, that one.

B

You just add a log there and it will automatically then start publishing logs pushing out logs. You can then do cube cuddle logs, get in look at each pod and look at all the outputs of each pod or sorry about that or you can just run this command, which is it basically gets all the pause in that are called core DNS and it will show you an output of all the all the pods that are running inside your cluster. So this is generally where you want.

B

You will find that most of the problems can be can be tapped into, and you can then act upon it. Some of the ways of fixing these problems and the what first way is the easiest way to just scale your replicas and you can scale them 2 3, 4 5, but sometimes that still doesn't help. So for that.

B

The second option is a local DNS add-on that core DNS has and it creates a demon that runs on each worker node and all the queries that come from any of those pods are then sent to this local daemon set, which enables faster name resolution.

B

The next thing is the memory requirements for core DNS, so cord e and s comes with some same defaults, so any case we've. We say that at minimum core DNS should have a 70 mid request and we've kept it to 170 mids. But you, if you're running a larger cluster, if you're running hundreds and thousands of pods, then you should change it and the calculation is pretty simple.

B

Pods plus services by 1,000 plus 54 in some of you may be thinking why 54 or 54 comes from 19 meg is what the core DNS application needs and then 19 in then 30 mix for caching and then five Meg's for buffers. So that comes out to be 54. That's the default! That's a recommendation in eks you'll be using 70, but if you're running large cluster you might want to, you might want to customize this number. Another plug-in is or the auto path plug-in, auto path plug-in.

B

It has some improvements that can improve name resolutions for names that are external to your cluster. Now I wouldn't recommend and recommend this enabling, by default, unless you're running into this problem of external name resolution. Sometimes, with Auto path plug-in, the increase could be 10 folds.

B

So it's really important if you have that requirement in your cluster, but the downside is that now you have significantly more memory that your coordinates pods need so instead of possible services, but by 1000 over you here, you're dividing that by 250 so 4 times more memory requirement with Auto paths. So that's the networking section I'll give you back.

C

I'm not done yet.

A

We're not done you're not done yet. It's all right, cube, Caudill, so cue puddle is the primary way by which we interact with the kubernetes cluster. How does Q kernel work? So, let's try to understand the basic concept behind cube card. Oh well, of course you need a cube, config file. You know that's sort of the way it looks at it and default location for Q configures in your home directory in a dot cube directory in config, and that's where your configuration lives and what can happen over there is you know in that configuration file.

A

You potentially have multiple clusters that are configured and for each cluster there is a config section and that you need to refer to could be a local cluster or could be running on AWS or any other cloud or on-premise. Then you can use the commands in cube. Kernel config use context, specify the context that you want to use and use that to talk to that particular cluster. So that's the way. Well, of course, you can always look at your cube, config, environment variable so oftentimes. We have seen that hey.

A

You know what I'm trying to connect to the kubernetes cluster, but I'm not able to connect to the kubernetes cluster, so the tip over there is make sure you don't have cube. Config environment variable point it to somewhere else. You know because if that is pointed, and you still have dot cube config, it doesn't matter. It's going to take. Look look at the cube, config environment variable. It basically communicates with your API server. Once the cube cuddle is configured, then it talks to the API server.

A

All the commands are given over there there's a beautiful post by somebody working at github. You know where it talks about how does cube quadrille, really work and I have a link after the end of the presentation it talks about all the details. What happens when you say cue cuddle, create F deployment, it's a long process. It's a lot of things that go behind the scenes. If you look at it on the client side, for example, there is a client side, validation that is happening.

A

Then it infer your generators hey by the way you're trying to do a deployment, so I have a deployment generator, because the deployment generator knows how to look at the deployment spec and create the rest representation of site. Then it creates a runtime object and that runtime object is basically using. That generator is going to create your rest representation.

A

Then it does the API version, because you may be using a specific version of deployment and then it says: hey: does your API server really provide that API version and during the first run of you, know your cube cuddle by connecting to the API server it downloads? All the API is support and puts them into your local cache. So you can look at that. You know if you go to into your cube config directory, there's a cache directory over there as well. So take a look at it.

A

It kind of gives you the complete API versions that are supported over there and that's what it refers to for local optimization. Eventually, your rest request is created, which is the rest request, because end of the day is an API server. Rest request is created, authentication is done and the way authentication is done for eks is a bit different. I will talk about that a little bit later, but now the request is off to the API server a lot more things happen on the server side.

A

Of course, you know, authentication authorization happens, you're trying to create a deployment. Are you allowed to create a deployment you're trying to delete a service update a service? Are you allowed to do that? Then admission controllers get in very various, say, oh by the way, if you're trying to create such as a deployment, how many resources are allowed, the namespaces etc what kicks in over there. Then it really DC realizes the HTTP request. Just the way there are generators.

A

There are objects on the server side which deserialize the request and persist it into HCD initializers kick in and then your controller kicks in and all these controllers are doing is essentially the deployment said. Replicas set, the scheduler controller is they're looking at your HDD state and they say: oh, you want to create a deployment. So let me take a look at it and deployment means you want to create a replica set. So I'm going to create a replica set persist that into XE d.

A

Then the reply replica set per controller gets up, says: I'm gonna create a pod, so it purses the pod in it. Cd nothing has been created. Yet this is the time when you're, looking at your cue card, L get pod. This is container creating all this is happening behind the scene. For you, then there's a cubelet which is sitting on your node, essentially, which is querying your api server every 20 seconds. Looking for it hey, you know: what is there anything for me to deploy? Is there anything for me to deploy?

A

Because that's where you know scheduler has already your worker nodes picked up where the pod needs to be deployed, and now the cubelet kicks in that? Oh there's, a part that is attached to me. That's scheduled on me. It brings it up, identifies the CRI and then you know it gets. Cri is a darker runtime, for example. Then it uses C and I gets an IP address.

A

Then it pulls the images and all those things, and then it eventually lays out all the file system on your local worker node, and then it says hey by the way. The container is up and running over here. So I just want to highlight the complexity of the process that goes behind this anything and everything can potentially break here and that's where you will see different status and we'll get into the pod lifecycle a bit later.

A

But let's look a little bit further, so one of the common errors that we have seen from our customers is they say: hey you know what I'm trying to do. Q cuddle, get serviced and saying the server doesn't have a service type resource or resource type service service is a standard kubernetes object. Why is it doesn't support it? The error is misleading. How oftentimes you know you see a good error and say: oh, this is excellent error. Now I can debug it.

A

So what's really happening here is you're at least an ek as case your cube. Caudill is not authorized to talk to kubernetes cluster okay, so the first tip that we always look at the users is, you should look at. You know what version of cube cartel that you have that you have so say, cube Caudill version. It gives you the cube Quarter version and in kubernetes there is a catabolic. There is a the amount of difference between cube Caudill CLI and the API server is predefined.

A

So make sure you look at those API versions that you know they're in like not more than two versions. Apart then also update your AWS I am Authenticator in AWS LAN. The security is all done. Using I am okay, so if you have just like you know, just a cluster configure their you can't directly talk to the cluster I mean that cluster configure is generated by AWS CLI. Then it's okay, but otherwise the way it happens is from cube. Cardle.

A

You know if there is a I am role that you need to have that I am role behind the scene. Cube model invokes AWS, I am Authenticator, which then talks to the SD service, which gets a token. Then it gets a token and that token is then passed on to cube Caudill to go talk to the kubernetes cluster behind the scene. So make sure your cube, cartel and AWS I am Authenticator is to the latest version. That's the typical source of error that we have seen oftentimes. What we have seen is, you know.

A

Maybe your config is not up-to-date because hey somebody else created a cluster and they did not give me a config or they give me a config which is out of date. Now, if we have the AWS CLI, then of course you can use AWS eks command over there to create a cube config over there.

A

For you, you need to have the I am role and the cluster endpoint, but once you have those data, then you can create your new cube config for you or if the cluster was created using eks cartel, which we said is the official CLI, then eks Caudill also has a tool here and the commands for generating your cube. Config file are specified over here. So, for example, you could use AWS eks update, cube config. Another command is update, but essentially it gives you a new cube config that you can talk to so oftentimes.

A

What I would do is I would create this command not generate this into my cube. Config file I will generate it into a separate config file, set up my cube config to that config file. Again I'm trying to isolate the problem over here and then say now: go talk to the cluster. So can I help you debug your problem and isolate your issue and then in case of eki scuttle, then you can say: eks cut all utils right, cube, config, give me the cluster name, and then it generates the configuration file.

A

You have to specify the I am role as well.

A

Another thing that you wanna look at is: is your cluster even accessible, so you can literally give the command curl Kay HTTP and the cluster API endpoint, and it shows you that hey this is what is available now eks cluster uses I will just that uses. I am authentication essentially. So if you were to give this from your desktop, that will not work. So this command was really given from an ec2 instance that was set up in the V PC the same V PC as the cluster was created, because then you have this write privileges.

A

You know to be able to talk to the IP API server endpoint, and then you will see this response. If you were to do this from your server, then you will actually get authorization error message. Not hey. I can't talk to the server, because first of all, you are using HTTP and the server has to be HTTP.

A

Even if you give it a HTTP endpoint, it will not work so the way you know this would this could work is potentially you could use like AWS command to first of all, take the I am token eks command, say: there's a AWS eks get token command to which you pass it. A iya, I am role or Arne of the I am role. That'll. Get you a token, then you use that token.

A

Then you pass it on curl that hey use that token to make that request, because end of the day, that's the token is looking for when it's trying to do the authentication and then your command will work. So my point is: there are a lot of ways by which you can debug. If your service is accessible or not.

A

Another error source of error that we have seen is somebody else created a cluster and I'm using my I am role and is saying I'm not allowed to access the I am role so the way it looks at it is in the kubernetes cluster. We have a config map called as AWS auth config map and in the config map, the users and the roles that are allowed to talk to the cluster and at what different auerbach capabilities are listed over there.

A

So what you want to do is check be with your admin or the person who created the cluster or that hey that's my role or the I am really exist in the auth config map as well, and if not, then you want to get it added over there. Actually. So if you look at it at the bottom of the screen, there is a map roles and map users, so make sure you are added right over there.

A

Now you may wonder this is not the right Arn format it should be, are in coolin, coolin coolin I am, but the colon colon was kind of messing up our formatting, but make sure this is not syntactically valid, but that's what it's supposed to be so again, as I said, you know, if you are not part of the AWS auth config map, then you will say hey, you know you must be logged in to the server, so you are unauthorized and it would not let you communicate with the server.

A

So in this case you know what you're done is you'll get your own added to the AWS auth config map and voila. Now your cube Caudill works and then you can say cube model version. It will give you the client version, the server version or you can use Cupid cluster info. Then it gives you the server endpoint and what more details about it? You can get the component status and it gives you scheduler controller manager, HDD status, all that, so all those good things come along to you.

A

Something else we have seen is that hey I'm trying to do Q cuddle get nodes. It's not even showing me the nodes. This is one of the capabilities that is there as again I'm using Amazon eks as an example over here, but make sure your API endpoint is publicly accessible because oftentimes our customers ask that I don't want my API server endpoint to be publicly accessible. I only want it to be accessible from within the VPC. So this is a snapshot of the AWS console, so you can take a look at it at the bottom.

A

It says public access is disabled, so you need to select enable and then the public endpoint or the API server endpoint is accessible publicly, and then you can talk to it from your desktop a century. So that's an important aspect.

A

So some of the ways by which you need to look at you know what is happening in terms of cube, cuddle, to get work or to just be able to talk to the server, because, if that's not working, then no parts are gonna, get deployed over there at all and I'm doing this one as well all right, pods. All this work, all this creation of cluster. All this cube kernel to work. All this networking to work is to get that one pod running.

A

Let's look at pods, so in terms of pod lifecycle, there's a community's documentation is awesome, so it very clearly explains what are the different phases of pods and what are the meanings of that so I'm not going to get into the details of it, but essentially appending running, succeeded, failed and unknown are the five stages. The only happy stages are running and succeeded.

A

That means you know that the power is running well and even in running, depending upon how many containers are in the pod, it will tell you one out of one of 2 or 1 out of 2, so you wanna pay close attention to those numbers, because if it's one out of two, then one container in that part is not running, you wanna make sure is X out of X. No, it should not be X out of y, where x and y are different numbers.

A

Succeded means you know, you are probably running a job and then the job has completed and is succeeded. So that's good as well. Pending we talked about some of the reasons you know why it could happen. I'll dig a little bit deeper into that unknown typically happens when the cubelet is not able to communicate with the server network partition.

A

Multiple reasons that could happen and failed is again could be the reason where we have seen most commonly happening when you're not able to pull the image, for example, and that's where it starts failing so then again, you kind of look at the details of the power that hey give me more details and it tells you that the image not found and all those error messages start showing up for you.

A

So what we're going to do is we're going to take the fantastic hello deployment that you created and we're going to create 8 deploy, 8 replicas of that so I.

C

A

A simple deployment here or we scale an existing deployment to a trap like us and I, see that only four replicas are available. Ok, so again, not a happy use case. So let's take a little bit more detail on why pod could be pending and what are the things that we could do basically for it to work around that now, when I look at say list, pods and I say Keuka I'll get parts, it says, yeah sure for pods are pending.

A

What are some of the reasons why the pods may be pending? The most common reason is not enough resources in the cluster on a part you can specify resource limit, port and resource limit could be for CPU memory, so there is most like the most common reason. Is there's not enough of those resources in the cluster?

A

Ok, so that's one part of it, something that Ray already covered is you know not enough IP addresses in the cluster, because the way we allocate IP addresses is each a and I can give you certain number of IP addresses, and you do that sophisticated math formula, and then you say hey: these are the IP addresses available, so you need to kind of scale your cluster. If you wanna run sort of number of paths. Okay, you also wanna ensure that your all nodes are healthy.

A

So maybe you think you have a four node cluster, but really what you have is a two node cluster and the other nodes are not getting recycled for some reason, so make sure that their health is looking good okay. So now what I'm gonna do is I'm gonna say describe a pod, I'm gonna take a look at the one specific part that was not healthy, so I'm. Looking at it, it says, failed scheduling is the reason for that, and it says: zero out of four nodes are available.

A

Insufficient CPUs, as I said on a part, you can define resources, limit resources and limits, and those are for CPU and memory by default and we'll get into those those part of it actually next. So the important part here is these events, even though I created a deployment. These events are only available on pods, not on deployments replicas at a job because end of the day they are creating a part. So you really need to say describe that part and then you'll get all the details about it. Okay, so now what I'm doing is?

A

Okay, get me all the events, and we looked at that detail already. It says all the events that I'm getting is zero out of four nodes are available, so there is not sufficient CPU available for me now. I want to only get warning events, that's important and again we looked at it earlier. We also looked at, say: fuel selector, one of the things I also want to highlight is I can actually get involved, object, dot, name so now I'm actually specifying the specific pod name here. That show me all.

A

The events warning events for this spot- simple tips, I, don't know which can hopefully will go a long way and kind of debug in your problem and again we are purely using cube model. We are not even gone into any of the system tools, yet another tool that we have found handy is again cube. Curl, Gary, Vance show me, you know list sorted by timestamp, so I want to look at the recent event. You know flossed five last ten last twenty, whatever that number is, you can start specifying that.

A

Okay, so let's look at you know the memory and CPU requirements. What is the reason that my part may not be running so now? If I look at the describe deployment /hello, it says sure it is a nginx latest image that I'm trying to deploy over there.

A

But if I look at the limits, it says two CPUs as the limit and two gigabyte as the memory, and similarly the request is to, and memory is to a fundamental concept that you wanna understand over here is: the request is what it will initially get and limit is the maximum value that it will go to okay. So that's the way you want to think about it. So essentially, what I'm looking at here is I need, if I'm doing eight replicas of the pod, then I need a whole bunch of memory.

A

You know some essentially I'm looking at sixteen CPUs and I'm. Looking at about sixteen gigabyte, hey I'm running a sixteen CPU cluster, why this thing is not running: I have 16 CPUs. Let's look at that more now. The default CPU request is 200 m and 200 m is think of it as the point to CPU and in terms of a V CPU, if you think of in terms of AWS or any other cloud provider and none on memory.

A

So if you create a power- and you say- describe the power to me and then it'll show you that requester is 200 m and nothing on CPU and no limit okay, so if I need 16, CPUs and I am running a 16 CPU cluster or why those 16 CPUs are not available to me now.

A

The way I can look at that is I can give a command called as Q corral top nodes and if I give that command, it shows me sort of the memory CPU utilization on my cluster okay, so I'm gonna give that command. But it says: oh, this service is not available, so the service is currently unable to handle the requests, get nodes, metrics gates API.

A

Now this is an API which gathers all the data from nodes and it gathers all the data used to be heap store, but now it's more Prometheus, so it gathers all the data from your node and it publishes it and that's where the data is being generated. So what's really missing, for me in this case is really a metric server and again this is a very standard upstream kubernetes tool. So what you can do is you can install the kubernetes metric server over here, just literally clone the repo and in the 1.8 plus directory.

A

There are a bunch of deployment files go ahead, deploy them to your cluster and once you deploy that, then your services up or at least the API, is available, and you can verify that your matrix API is available. So you just say: get API service, give it a number- and it says, yep, looking good available and running for a while.

A

Now when I say cue, plural, top nodes, I'm getting the right data, so it's reporting the codes, the CPU, so I said yeah you're running about you you're, using about 28 M, so about point zero. Two eight is what you're utilizing zero percent is being utilized, and it's using about 410 megabytes of memory is about two percent of memory for all of the 16 gigabyte that I have available. Okay, now remember all of 16 Giga 16 CPUs are not available to you. That's an important part of thing.

A

You are running the 16 CPU cluster, but certain amount of memory is required to run cubelet and certain amount of memory is run to run the operating system, demon, the CRI and all those things. So all of 16 CPU is not available. Let's look into that. Then there is a concept of capacity memory and allocatable memory, so the capacity.

A

No top nose is top nose is only for the nose itself yeah. The question is the top notes. Command is, is using the request and the limit. No, it's not top notes is just saying that how much memory and CPU is available for the nodes, correct, and so now what I'm saying is: there's a concept of a capacity memory and allocatable memory.

A

The capacity memory is basically how much capacity is available and allocatable memory is sure you have a capacity of 16 gigabyte, but certain amount of memory is already being used for cubelet and other processes, so I want to make sure that I'm giving you only the allocatable memory- and you can see there's a bit of a difference here is that here is about hundred one five, nine five, and it is one five, eight four, so whatever that Delta is, is being used for cube black okay.

A

The way allocatable is calculated as a matter of fact, you of course look at the node capacity, so you remove cube reserved memory. You know whatever is required by the cube lat. Essentially you remove the system reserved memory, and you can also set up the hard eviction threshold memory, which is the memory which you know at which point you know, it'll start ejecting parts. You know because it knows it's right of the threshold. Essentially so one of the key concepts is, this is explained at node allocatable.

A

So it's worth reading on how all this thing work the allocatable and the capacity memory, because that is a fundamental part on how you are designing your cluster okay. So now, let's take a look at it when I do the same thing for capacity and allocatable. If the numbers don't change, it says the cpus are four and four. If four is allocatable, why that means I have 16 CPUs. Why I'm not able to allocate it okay?

A

Now, yes, allocatable should be calculated that way, but when you are creating the cluster at that point of time, you as a cluster operator cluster designer, has to specifically say oh carve out this memory for cubelet reserve, and then it says I'm yeah, because you are asking now I'm gonna, take that into consideration of calculating the allocatable part of it.

A

Okay, so one of the things that we recently- and this was literally done about two weeks ago, when using eks now when you're creating a kubernetes cluster, we allow you to set those values, eviction hard and cube reserved, basically those two values, so those are set in the kubernetes cluster, and that is a clear indication that okay, this is the amount of CPU and the memory that is available, so those are allocated for system processes.

A

So now it's evident that hey, we don't have enough CPUs. So what we need to do is set up a cluster autoscaler cluster autoscaler is again an upstream component. It serves two purposes. It basically makes sure that if any parts that are failing due to insufficient resources, then it'll just scale the cluster. For you and then it will also recycle nodes that are underutilized for an extended period of time. So let's say you have a 10 word cluster and it recognizes that hey by the way, all the nodes are running about 20% of CPU.

A

So what I'm gonna do is I'm gonna, pods or straight lists, I'm, going to move them to a different node and I'm going to recycle those pods I wanna make sure that your cluster size is optimal.

A

So what we're gonna do is we're gonna, install it and again for installing it we need to setup certain I am policies attached. The policy to the I am role we on the auto scaling group which are behind the scene. For us, this is what is connected to so cluster autoscaler is a kubernetes concept, but end of the day.

A

They need to talk to auto scaling group, because that's where the in AWS language, where your cluster is scaled, so you need to make sure that you set up the correct tags over there, and these are Kate's IO tag. So once you set up those tags on the auto scaling group, that means auto scaling group can be automatically discovered by a cluster or a scaler.

A

Now you create the cluster autoscaler, so you essentially need a certificate file. Then you create the cluster or a scalar here. Okay, I need to look at it that ok, now I can look at my cluster autoscaler logs. Essentially, and if I look at my autoscaler logs, they are the parts of the cluster or a scalar are running, but it says skipping node group over here because it says the max size is reached sure I've installed the cluster or the scalar, but as I said, the cluster or the scalars behind-the-scene talked to auto scaling.

A

So the next thing that we need to do really is update our auto scaling group limits. So we go to our auto scaling group and to the auto scaling itself. I, say: oh by the way change my Mac size to eight so now, I can set up my auto scaling group to be eight one thing to understand over here is the cluster or a scaler is being reactive?

A

In the sense that it looks at oh by the way the pods are not being scheduled and now I need to scale it, you can actually use auto scaling group use the auto scaling group policies to being proactive in which you can set up in the auto scaling group that hey you know if the CPU utilization goes above, 50 percent automatically scaled, my cluster and add more nodes to it. So you can use a combination of the two to make sure you know your pods are always like you know.

A

Your cluster is an optimal size and your pods can always be scheduled.

A

So now, essentially, if I run my cube, kernel get parts, you know, cluster a scale or a scaling group is set and all my parts are running. Basically at this point of time, similar use case I was running a machine learning workload on this kubernetes cluster over there using cube flow and I saw that hey m missed inference, I'm running an inference task here and I say zero of one ready pending and let's describe this part a bit more and as I described it, it says all right. You know insufficient NVIDIA GPU available to me.

A

So essentially is my solution for that is set up. My cluster autoscaler use the auto scaling group to look at that matrix and say what scale my cluster and that way it can solve that problem.

B

Awesome so so now we have very easily created a cluster. It was super simple right. We didn't run into any problems because we read the documentation first, but now you have a pod, add hello, world deployment running and Nick. We want to show the rest of the world, how cool it is and how you know the amazing fonts that we've used in the CAD pictures that we've used. We want to show it to the rest of the world.

B

So a couple things you can do is you can use a tooth either the service to expose your paths to the rest of the world so then will be accessible from outside your bluster or you can use the ingress object. So if you just do service type load, balancer, if you just say, expose my service, then it's going to by default install create a classic load. Balancer. We have three types of load: balancer, Network load balancer, which is there for classic does four and seven and an application Allah balancer. That does there are seven.

B

So if you, but if you just say, service type load, balancer in your service, then you're going to get a classic load balancer. If you wanted to use a newer, Network load balancer, then all you have to do is annotate your service with just load, balancer type, NLB and e KS is going to provision and NLB in the backend. Now you don't have to be using eks for that you could just be running kubernetes on AWS and and then that should work as well.

B

The second option, if you're operating in micro services environment, is to use al being rest controller and it creates the ingress resource inside your cluster and then you can basically say when you get traffic on slash front-ends and send the traffic to this particular service. That's running front-end and when you get the traffic on slash back and send this to another set of service and eventually to the pods that are running the backend service, so you can use l being pressed controller. You just install the controller inside.

B

Will provision and al be on your behalf in your account and that's it? Then you start creating your ingress rules and you can start serving traffic. In addition to al being rest control, you there are a bunch of open source and commercial ingress. Ingress is out there. The most popular one that I've come across is nginx, so that the really works really well, but it doesn't mean that you're, you know creating services.

B

You know you get the theme right, we're not gonna just say, create, load, balancer and automatically it's going to happen because we're the worst users. Of course it will fail. So when it fails, if your load balancer is not provisioning, the first thing I would do is get checked tags. So there are, if you go to the documentation hidden in very deep inside our documentation. Is this really important tag that you should put in your subnets?

B

Otherwise, your load balancer is not going to work so make sure that you, you have those tags in your in your subnet.

B

If you still, if you have the load, bouncer enabled and you're still it's still not working, then the next step would be to go check your service and check the events check the status, sometimes we'll say, load balancer provisioning failed and it can give you some interesting idea of what to do next. The last thing I would check is the eks service. I am role so remember. I said: when use they say. Service type load, balancer eks is going to provision a load balancer in your account.

B

This role gives us it gives us the ability to do that. If you don't have this role, then we basically don't have the ability to come in your account and start the create a load balancer so make sure you give that permission to us now. If you have your load balancer, it gets provisioned. Don't think that you're not going to run into some problems there. So it's the most common problems that I face is you know everything works, but my application is slow and this is without Kremlin's chaos. The engineering.

B

This is just my hello world. You know this cannot be chaos. Engineered I thrive in chaos, so first thing you should do very easy to do. Is enable logging inside your load balancers, it's very easy to do. It's a checkbox. If you're lazy like me, go in the console enable it that will give you some of the metrics like latency and application metrics like 4x, X, 5, xx, errors and those are helpful. You can see if there's a high error rate in your application. That could be the reason why it is it's.

B

Your request is slow or failing. If you see that there's everything is good inside your at you. Look in your load balancer front. The next thing I would do is start a part inside your inside your kubernetes cluster and access the service right there and see what the metrics are.

B

So you can create a shell demo pod and it basically gives drops you into a Debian or go into like environment, where you can install some really good troubleshooting tools like Carl, and you can curl directly the IP or the fully qualified domain name, and you can see what it really is that that's causing problems so a lot of times. You will see that the application is slow either, because your pods are spread out too wide nodes have latency in between them, and this is a good tool to help you identify now.

B

Another thing that you can do is when you create a service you're, also getting endpoints, so you can also try to ping the endpoints and see if one particular endpoint is is creating problems and the rest are fine. And lastly, if you, if you, if you want, if you can't change your app, definitely implements tracing in your application, micro services without tracing is super hard to do because you're, basically, the request could be coming in from one app and traversing through twelve different apps and then giving the result back.

B

So if you have latency in one of the app in the twelve apps that you have, you can't see that unless you have some kind of tracing ability and traditionally tracing has been really hard to him to implement.

B

But if you're using a service, mesh tracing becomes really easy and I would highly recommend, you use some kind of tracing solution with service mesh, whether it's a jogger, open, tracing with ISTE or x-ray, with the app mesh definitely use it, because it's low effort- and it will give you the ability to look at that request and see where it's failing or what is the what's calling the slowness in your environment.

B

So clearly networking really really important monitoring that really really important piece that you cannot skip if you have to run a successful kubernetes cluster so that you asked a question like ok seems like kubernetes. Has these millions of resources? No I'm? Sorry, it's a billions of resources. What should I really monitor, and that becomes a problem. It's too much to monitor and there's too much data causes too much confusion. You don't really know, should I be paying attention to this. Api latency metrics or this resync interval.

B

What is what is it really so we've tried to simplify it for you. You can of course, monitor everything and you should monitor everything, but you don't have to pay attention to every metric. Some of the key things that you want to monitor are your worker nodes. Clearly they are running your worker or your application so make sure that they're running healthy.

B

Then it's going to be a cluster of components like ingress controllers, dashboards tools for monitoring cluster autoscaler, all of those things and then finally, the apps, so some of them metrics that we recommend that you monitor to begin with on the nodes, definitely have CPU network memory. That's the basic cluster components at CD is something you should monitor if you're not running eks, because if you are using it. Yes, that's our responsibility. Amazon will make sure that your at CD cluster is healthy and up and running all the time.

B

But if you're running your own cluster highly recommended, because if you're at CD cluster goes, it dies and then we'll say goodbye to your kubernetes cluster advanced, really important, monitor the monitoring tools and then have monitoring tools monitor them. We must go level deeper. So if you have monitoring tools or ingress controllers, load, balancers dashboards, make sure you monitor those components as well, and the last part is your application.

B

You definitely want to capture metrics that are relevant to your particular workload and any of the error rates like if you're running a web-based application, 5 X X 4 X X. So how do you monitor? Well the simplest answer today in communities? World is Prometheus with Ravana, everybody loves it. It's super easy to set up, there's a there tons of community support, and so that's the de facto tool. A couple months ago we also launched cloud watch container insights.

B

This is, and it really easy to use tool that you can install inside your cluster and we have pre-built dashboards. That can give you a lot of details on your cluster or you know your pod level and there's certainly kubernetes dashboard. You can use that, although I'll stop there, then there are commercial solutions, a bunch of them dynaTrace data dog-sitting are just some of them. They can provide you some really cool functionalities, like some of them have machine learning that can predict problems before they actually happen.

B

Let's start looking at, let's look at Prometheus, so Prometheus again is very easy to install. It comes with a helmet art, so I would here I'll just created Prometheus, namespace and I'll say helm, install Prometheus now, I I do want to customize some of the values for helm chart and over here I'm, going to say that I want to persist the data, if you don't do that, basically you're getting a femoral pod which, if it just gets destroyed, all of your metric data will be lost. So you don't want to do that.

B

You definitely want to persist that data, so I've created a storage class, that's backed by a EBS, that's the elastic block storage and over here I'm, using a GP to type volume that I created before and I'm saying from easiest. Please store your data into this volume and then also I'm, saying for alert manager, which is a component on prometheus I'm saying you all. So please go ahead and persist. Your data next thing I want to do is once I create the the deploy the hunk chart is going to create a bunch of tools.

B

A bunch of pods inside my cluster you'd, see that I'm running a three node cluster here and for each cluster for each worker. Node you'll see a node exporter here and then you'll see Prometheus server, a alert manager and state persistent pod. So basically that's what you get when you install Prometheus. If you want to check for meteors you can you can I just do a cue card board forward which is going to open a port on your localhost, and if you go localhost port 8080, you will get Prometheus dashboard.

B

It's not really helpful, but you know operators are not supposed to forgive you information first hand. So if you want to sleep nice, pretty graphs that there are pleasing to look at- and you know, make errors, look better, then highly recommended that you install a graph on on top of it. So again, Ravana you can use ham chart to install graph on in your in your cluster. You don't have to create a namespace from graph on. A lot of people will just put this with coop system or prometheus namespace.

B

But if you want to, you can again I'm here saying that I want to persist. My data on a GP to storage class, just like I, did with Prometheus and then I'm I'm saying that I also want the service type to be load balancer.

B

So in this situation, what I don't want like Prometheus I, don't want to go and every time say to cuddle proxy or queue cuddle to open a port on my laptop, sometimes I may want to just access my cluster state outside my network or not for my laptop, where I don't have cube Caudill so over here when we say service type load, balancer, it's actually going to provision a classic load balancer in the backend that we can just go stable URL.

B

We don't have to expose any ports, no dependency on cube, Caudill and access brief on our dashboard. So again, once I deploy the helm chart you get a one pod, and that does everything, and the next thing I want to do is I want to I.

B

Want to get the URL so I'll, you just query the the graph on a service and just get the load balancer address, which is going to give you the ELB URL, and then you just get the the password, and then you get the dashboard. Now when you go to co-founder dashboard, it's not gonna. Look anything like this. It's going to be very basic. You can create your own dashboards. We recommend it, but you probably just want to use the community driven 1 & 4, because de fauna is not a kubernetes specific thing.

B

They have Redis the dashboards and they have Cassandra dashboard. You can't just go and reuse. Some of the community created dashboards over here we're looking at community dashboard, 31 31, very easy. If you go to Griffin, ax you'll just say import and you put in a number and you get this dashboard, you don't have to do anything. Another dashboard, that's useful, so 31 31 is a cluster level. It gives you pods and it gives you nodes and cluster level details 3146. It gives you pod level details.

B

So it's um and they're a bunch of a bunch of dashboards out there that you can utilize inside your cluster. So you don't have to do any. You know there's no learning curve with your fauna. You can you don't have to note how to create a dashboard, just get the number and use it in your cluster.

B

Some of the important log files that you should also look at are going to be on note logs your communities control plane, logs scheduler logs audit logs. These are really helpful if you are in in an environment where you need to go back in time and tell what happened or they're really helpful in troubleshooting, so highly recommend that you keep these logs, you can you don't have to store them forever, but you know, as long as you have to go back for troubleshooting. Those are good enough. Now.

B

I want to talk a little bit about the container cloud watch container insight so because cloud watch team has built this really nice tool that they think that will help a lot of customers. Traditionally, installing dashboards has been a little bit of pain point and then learning how to use that dashboard effectively in your organization is another feat, so we wanted to solve those two challenges, make it easier for you to collect, summarized metrics and then consume metrics at the same time.

B

So if you use cloud, watches really easy to use, you get some dashboards that are already built and all you have to do is just go there and monitor, and then you can put an alerting and trigger send slack message, all those things so a we. We use cloud watch, but on the cluster itself it's leveraging fluent D, which means they have all the logs inside your cluster, assuming that it all centers standard out are collected by affluent D and then fluently pipes or sends those logs to cloud watch.

B

Cloud watch is known, not the only destination that fluently supports. So in future, if you'd like to say I, don't like cloud watch, I want to go to elastic search. All you have to do is in that in fluently configuration you're, going to say now start sending cloud. It locks to cloud to elastic, search and that's it. You don't have to rip out your entire infrastructure there.

B

So in order to deploy cloud watch container insight, they all you have to do. Is you import a yamo that we provide and it's going to create two daemon sets. One is going to be the fluency demon set. The second one is going to be the cloud watch agent daemon set, and that's it once you do that next thing you open your cloud watch dashboard and you can. You can see this and that's just the one type of dashboard that we have.

B

We have created custom, dashboards and- and we spoke with customers like you and I said. What would you like to see as a first thing and a lot of customers said we need cluster level details, but I also want to be able to see my thing. My pause by namespaces I want to see my services and I want to be able to dig deeper into that cluster.

B

So, for for that, we created a bunch of dashboards that will Attis fire most of the needs out there, and you helps have the ability to create your custom dashboards. If you want it, another thing it does is logs. So along with metrics that's cloud watch is collecting. There are also logs that cloud watch can take so again we're using fluent D send those logs. Those is stored, durably in cloud wash so again, you're, not storing them in another location or the stored in cloud watch.

B

And then you can use cloud watch, logs insights, queries and then query those logs without having to provision any c2 instance or any hardware. So, for example, if I wanted to see what is the memory usage across my cluster but I don't want it relogin to cube cuddle. I can just run this query on my logs and it will tell me what is the current state of my cluster. So there you go without provisioning, an ec2 instance without provisioning. Any hardware you can query logs very very easily.

B

So, if you're, if you, if you're running an application, if you're creating a micro services, environment, monitoring and logging are really really important, a good and cloud native application is not only just that what that runs well, but also is it's also reporting what is happening? You don't want to be running a black box. You want to see what is running. What are the metrics? How is the application running? Is it failing or not so for that there are certain things that you should do.

B

First of all, probes make sure you use probes as default, make sure you have liveness and various probe as a standard across the organization, because they're really helpful, you don't want to be running into situations like your pod comes up and it's it's not functional for five minutes, and then your cluster is getting auto scale to millions of pods, because you did not have a redness readiness probe so make sure you have them: implement tracing use, service, mesh or use x-ray, open tracing whatever is a tool you love and then also monitor, process, health and logs.

B

So don't just don't just assume that, because you have flu in DM clogged watch that all the logs are magically going to appear in cloud watch. Make sure that all the components that you are running, for example, if you're running a Python app that uses nginx, make sure you capturing both Python law, Flass, logs and nginx logs, so make sure your all. The logs are being collected, not just your app, but also supporting apps that you may have.

A

We can I talked about this a little bit earlier already. You know in terms of how do you do resource reservation, so you can monitor the cubelet on the worker and also, let's say you have done the resource reservation. You can dig into your worker node.

A

So if you create your kubernetes cluster and let's say you have ssh key enabled over there, then get into your kubernetes cluster and say give me the details for a particular worker node, and this is the way the way you are getting it, and we talked about this earlier already how we can use cube, reserved and system reserved memory to how this value can be set up, and not only you can do this like if you're running a eks cluster. Of course you can use eks cut all over there, but let's say you are creating.

A

You know: kubernetes cluster, you know using some other tool. So cops has these capabilities as well or if you are creating using army, then in the army there is a concept of a user data and in the user data, for example, you can specify these commands as part of your commands to the API server, and then you can say that make sure my resources are appropriately allocated.

A

So, if you look at it over here, for example, we are highlighting that how you can set up that value in the eks coddle, and this is only specific to Amazon eks.

A

Essentially, so that's one of the areas where we are talking about so how you can have system reserved, cube, reserved and eviction reserved, because those are the three values that are taken into consideration when it's looking at capacity and actually reporting the right value of allocatable, so that, yes, you know it will not be able to schedule your parts, but at one you wanted to report the right value as well for CPU, okay,.

A

If you're doing, we have seen a lot of customers who are doing oversubscription where they say you know what you are sure. No CPU is not a compressible resource, but memory is a compressible resource, so I'm going to over subscribe it. So one of the things that we recommend to our customers is to avoid over subscription to specify sort of your CPU and the memory limits over there, and in this case, for example, you can see that the memory requested is 64 megabyte for the memory and 128 megabyte is the limit.

A

So that way, you can totally calculating on how much max that the each worker node should be able to handle another option over there. Another opinion over there is to be able to specify your resource quotas or namespaces, so in this case I'm just saying that, for this namespace and or depending upon your dev and the testing and the prod and the staging namespaces, if that's how you are achieving multi-tenancy in kubernetes setup your limits over there and it's not just for CPU as a matter of fact, you can set it for CPU.

A

The total amount of computer resources, the total amount of resources, how many services, how many parts can be created in this namespace. So once you start specifying that, then those- and there are inheritance rules available that if your pod spec does not specify any resource and limits that it inherits that from the namespace itself,.

A

Now, once the namespace on the coton, the namespace is full, of course, I know it's treated as the same way that hey you know, you no longer have any more resources. So then, your cluster is not the scope. Now, your namespace is the scope. So then you want to look at it that ok, the error message is, fortunately, are much better here. It says, exceeded quota and it will tell you exactly what quota is exceeded and then you start debugging it. Then you go back to your cluster operator.

A

That increase my namespace limit will give me a different namespace, a different cluster, so that you can start debugging. What's the problem over there now by default, containers run with unbounded, compute resources on a kubernetes cluster or in our namespace resource quota. So limit range is a policy to constrain resource by pod or container in our namespace.

A

So in this case, for example, I'm creating a limit range and I'm saying that by default, if the pod has no spec like no resource no memory or CPU specified over there, this is the memory that they're going to inherit. So that way, you can start sorry.

A

They're gonna inherit, basically so that kind of gives you an idea that you know how you can start sizing your cluster, because cluster sizing is really important. You know you don't want it to grow arbitrarily large.

A

So now what you're doing is you're just saying: keep Carol described limit ranges, and then you can start getting those values and that kind of gives you an idea that you know why your cluster may not really be getting created. So from your perspective, you're just saying cube, Carol create deployment, and then you have to recognize that hey. Is there a namespace limit? Is there a limit range defined for me? So then you have to start looking at those objects and their default values. Kind of the reason that the part may not really getting deployed.

A

I'm, not gonna, dig in deeper into this, but essentially the idea is if there are rules, if limits and requests are defined on the pod and on the limit ranges and how they really work. But the whole idea is that, if nothing is defined on the pod, then they inherit from the limit ranges, that's sort of the key rule over here, but once again, the key part being if your part is not getting deployed. If your part is not coming up, there are things possibly going at multiple levels.

A

You know it could be limit, rain could be namespace, coda could be cluster autoscaler, we haven't even talked about as a matter of fact about horizontal part autoscaler, and that is one thing that can help you really with with the service latency, which break kind of touched upon it briefly,.

B

All right story time so I, very proud of creating my hello world application, but I also create another application where I can still store. All my important data and pictures so I took the my my kids, birds, pictures and baby pictures and stored all in a kubernetes pod right, it's awesome and then the pod obviously got deleted. So guess what happens to all my data? Anyone any guesses, the data was gone. Yeah I was kidding. It was not my baby pictures when wedding pictures. Much worse. Trust me so pods are ephemeral by nature.

B

Do not store your wedding. Pictures. Do not store your tax documents in pods. Unless you are using stateful sets, then you can do it so stateful sets allow you to persist data using storage back-end that again is abstracted away from the developers and users. You don't need to be. You don't need to care as a developer. What storage back-end should I use? Those are all done by hopefully an administrator.

B

If you're the administrator, then it's you, you should do it your job, so stateful sets allow you to persist that data on a on a on a back end and it will container if a pod fails, then the data is persisted and the new stateful set container would come up. Pod will come up and start using that data, so you don't lose any data, so stateful sets are really providing pods.

B

With that persistent storage to persist data, they also create percent volume claim automatically on the fly and the reason why stateful sets are so different from deployments and pods are because there they are. Each pod is going to get its dedicated PVC, which means that it's not sharing that volume with any other pod. It will have its own dedicated PVC.

B

It also creates a headless service, and what headless service really does is it gives you each each pod? That's part of that service, a unique IP address, so, if you're running, let's say Redis master and Redis slave, you can really easily identify the ordinality, so like one would be. The master second would be the slave and so on, and that ordering is maintained by by kubernetes. So if your pod or your your node fails, if multiple nodes fail, when the new pods come up, they are going to come up in that same order.

B

So it's kind of a little bit more resiliency that you can put when you're working with stateful sets, which is really necessary. If you're dealing with databases, for example, you may have a my sequel, master database and a bunch of slaves. This is going to make sure that you're always replicating from master to slave and not otherwise so an eks. There are a bunch of options. Some of them I, wonder highlight here, are the EBS and EFS CSR drivers now CSI is a container storage.

B

Interface is much like networking where you don't where you can pick your own network provider, you can use a dub Amazon, VP, CC and I, or you can use flannel. Similarly, for storage, you don't have to use the same kind of storage provider. You could use EBS, you could use EFS. There are a bunch of third-party providers out there that you can also use again they're big.

B

What they conform to is the CSI spec and all of them provide the basic functionality which is providing a volume, and then you have a bunch of additional functionality like snapshotting, etc. So, if you are running databases, if you're running pods that need to process data, make sure you're running stateful sets don't delete. Your wedding pictures make sure you're reusing one of these CSI providers now period a few generations ago in kubernetes there was an entry, a storage provider, highly recommend that you don't use that, because the community is moving away from it.

B

You want to use something that provides that's using CSI. So if you look at ETS, EBS is the most common most common storage type for customers that are running stateful sets in kubernetes. So you can use that with eks or you can just reduce it with cops. It doesn't really matter, but they will. They will provide that persistent storage for your pods in EBS.

B

These are again the functionality functions of the specific CSI types of EBS CSI type support, both dynamic and Static assignment of volumes, which means that either you can pre create a volume in EBS and say, use this volume to store all data, or you can say, as a pod comes up just dynamically, create because I don't want to be managing volumes on myself, and you can also say that if the pod goes down, if the data is replicated delete, the data as well so you're not paying anything any additional for for that pod, that's not alive.

B

These volumes can be resized, so you don't need to guess the capacity ahead of time. You can just increase from 10 to 20 GB if you wanted to, and they also support snapshots. So you can take a snapshot if you're not running a pod continuously. Take a snapshot and abs kill the pod. There are two months later when you need it again, you can start a new volume and hopefully save a little bit on cost.

B

One important thing really really important thing with EBS is currently EBS is only available in EBS. Volumes are only available in one availability zone, which means in a region like us west, to where we have I think three availability zone zones. If you create an eps volume in availability zone, one it's not available in two and three. So if you're, creating stateful sets or if you're, giving your pods a volume, then make sure that you have some kind of logic.

B

That says this pod can only come up in this availability zone and that's typically done using node labels. So make sure you don't run into situation where your pod is now was running in us us West one and was using an EBS volume in that availability zone. It got deleted and now it's running in West, two waiting for that volume, which is never gonna arrive. So don't don't get run into that problem, make sure you're using your binding. Your stateful sets. You make sure your binding, your pods that need storage to a particular availability zone.

B

Now, if you wanted to, if you wanted your your pods across availability zones to share the same volume, use EFS EFS is available across the region, so it doesn't matter which AC you sit in all the volumes are replicated across. So that's a really good use case. If you, if you need that kind of distributed storage, creating an EBS storage class is really simple, you will say provision, or this is going to be AWS EBS and over here you can provide some parameters.

B

So here I'm saying give me a general purpose volume, and this is going to provision EBS volume based on this configuration here and I. Don't have to go to EBS and say give me a volume. I can just create everything in my kubernetes cluster and then, when the pod needs an idea volume, then storage class is automatically going to provision one assuming that it's doing it dynamically and attach that to the pot. So it becomes very seamless.

B

You only need to create a storage class once and then the developers can just request that source class and request volumes for their pots.

B

So one of the things that I, actually let me step back I- was trying to look for failure. Scenarios for storage didn't really find much, which tells me either. My google searching is really bad, or these csr drivers work very well I'm, proud of my google searching skills. So that's definitely not it. It looks like it's a lot more stable environment now, but there are certain things that you you may run into. One of them is this, which is you mount a volume, and you now want to write, write on it and it fails.

B

You know often customers, we're gonna, ask me and say what, if I do this? This is a crazy thing in kubernetes. Is it going to work, and my answer is in Spanish see, but in Spanish see also means. If so, my you know, they misunderstand me. They think I'm saying yes, which is C, but what I'm really meaning is, if, if you read the documentation right and if you do everything right and jump to 200 hoops, it's going to work.

B

So this is one of those scenarios where you may have mount a volume and you can't write to it. The reason why it happens is because, if you're really interested Google search, pause, containers and they'll tell you a lot give you a lot of reading to do, but all the volumes are mounted as by a root user. Essentially, so, if you're not the root, you can't write and don't run your containers with root.

B

That's not what I'm trying to say, but what you can do is you can either use the security context in the pod, which is going to use Linux security context and is going to allow your containers or docker container in this case, to assume that security context and start writing, or another thing you can do is in it can so a lot of people will say in it container, which is which runs, which you can run as root mount the volume and then change the chain.

B

The permissions which is Rancho noir chmod, or something like that to now give your other containers which really need to write to their volumes. The ability to write to the CVS volume, so that's those are two things you can do if you want to put security context. Basically, this is what you need to do.

B

There's a little bit further reading required here if you're not familiar with these two things, but if you're dealing with storage- and then it's highly recommended that you that you implement this, if not, then in it, containers could be a route now in containers, not they're, not just for storage but anytime. You need something done before your application comes up, use your use in it containers you can use ordering there and they're really helpful too, and that's it for my site.

A

So this presentation is all available online. Now this is the github repo on the top, where all the content source is available and on the bottom is where you know I mean we have been running a local version of it. But if you go to the bottom link, you can actually run this wherever you want to so feel free to share contribute. We would love to hear your stories on what happened to your part. Why did your part fail and I?

A

Think I'm gonna write a thesis on that sometime now, you could actually very easily write a story on this end. The pr yes.

B

In a PR, you can edit this presentation yourself, yeah.

A

This is all on a github repo. Some of the links that we want to share here is the last link is what I want to really highlight. It's: a simple link gates, dot, a fk8 s, dot AF. This is a link of kubernetes failure stories. You know the multi-million ways your kubernetes cluster can crash and burn. So we all learn from our failures for more. We have some books that we want to share some of the books that we like and that are well used in the industry and that's a wrap. Thank you. So much.