Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond Block Diagrams: Different Ways of Understanding K8s Architecture - Kim Schlesinger

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Beyond Block Diagrams: Different Ways of Understanding K8s Architecture - Kim Schlesinger, DigitalOcean

Most introductory Kubernetes videos and blog posts use static block diagrams to illustrate the core components of Kubernetes and how they interact. These diagrams don’t convey the dynamic communication between the K8s control plane, nodes, and pods, and we owe it to ourselves and people new to the field more accurate and engaging ways of representing Kubernetes architecture. This talk will demonstrate a few different methods to visualize how Kubernetes components work together including a time-sequence diagram that models what happens when a user creates a pod using the Kubernetes command line tool, kubectl, and how to use Jaeger to see an http request move onto different nodes by traveling through a load balancer, to a service, and arriving at a specific pod. This talk is for people who are new to Kubernetes and want to understand how the system is designed, as well as experienced engineers who are curious about alternative ways of visualizing Kubernetes architecture.

A

Items different ways of understanding, kubernetes architecture. My name is kim schlesinger and I want to tell you a little bit about my career and how I got to the tech industry.

A

Currently I am a developer advocate at digitalocean, where I focus on cloud native technologies before this job. I was a site reliability engineer for a few years. Prior to that I was a javascript developer and my first few jobs out of college. I was an elementary school teacher teaching here in the united states.

A

I bring up my career timeline to tell you how I got into tech. So when I left the classroom- and I was looking for a new career, I eventually landed on wanting to write code wanting to be in the tech industry. So I attended a coding bootcamp where I learned front end and back-end javascript and then became a web developer for a few years.

A

Although I enjoyed web development during that time, I realized I was really interested in ops and devops and system administration and how do applications get on the internet and how do they become accessible to users? And so I eventually found my way into a job as a site, reliability engineer.

A

I started that job in 2018 and I was working at a company called fairwinds, which is a kubernetes consulting agency, and so my first set of experiences as a system administrator and as an ops person was as someone who was a kubernetes administrator, which was really really tough.

A

So there are a lot of people like me who start in the tech industry, and they are already working immediately in the cloud and I'm going to refer to these people as the cloud native generation cloud native generation is anyone who starts their career by writing software and deploying it into the cloud, and it can be really difficult as someone in the cloud native generation to really fully understand the systems that we're working with, because it's very hard to build mental models of things that you can't see and touch.

A

So we all use mental models to think about and interact with systems like kubernetes and a good definition of mental models is any framework that you hold as a mental representation of an external reality.

A

When I started at fairwinds- and I had more senior engineers, mentoring me and trying to help me understand, kubernetes the system itself often static block diagrams would come up. We've all seen things like this and they are really helpful, diagrams and representations of kubernetes, but they're most helpful. If you understand the technologies and the practices that kubernetes is replacing and automating, and since I didn't have that prior experience, these block diagrams were extremely difficult for me to understand, and I didn't find that they were helpful.

A

So to get you thinking about this like okay. Maybe these block diagrams describing kubernetes are helpful for me, but they might not be helpful for people that we're hiring. I think it's really important to be thinking about this, because we want to be hiring training and retaining early career engineers who have limited computing experience. People like me who didn't grow up with computers who didn't get a computer science degree.

A

We can still be excellent engineers, but we have to be a little bit more creative and a little bit more deliberate about how we help people understand the systems that we're working with, and so in this talk, I'm going to be showing you three different ways of visualizing and understanding kubernetes clusters that aren't static block diagrams.

A

So if you're, someone who's mentoring, more junior engineers, these are the takeaways that I hope that you'll have the first one when you are creating a representation of a kubernetes cluster create diagrams that show the passage of time. The second type takeaway that I hope you have is that you should use distributed tracing tools as a learning tool, not just as something for identifying and debugging errors and then, finally, you should build 3d models of kubernetes clusters, so people can actually touch and feel something and develop a richer mental model of the system kubernetes.

A

So in order to show you what those takeaways might look like, I'm going to give you three examples of alternative ways of understanding, kubernetes architecture. The first is a time sequence diagram.

A

The second is combining distributed traces with a diagram, and then the third way is to actually show you what a 3d model of a kubernetes cluster looks like. So, let's get started with the time sequence diagram.

A

So the example that I'm going to be demonstrating is you have to imagine that I'm a kubernetes user and I want to create a busy box pod. I have this pod manifest on my computer. It's a yaml manifest and I'm going to run the command cubecontrol, create f, pod.yaml.

A

Here's the time sequence, diagram for that sequence of events.

A

So, let's go through this so on the left. There's me kim and I'm running that cube control, command, cube control, create dash fpod.yaml, and that request goes to the api server immediately. The api server goes on to step two where it saves the new state in etcd and then the controller manager is checking the api server for changes.

A

The scheduler is also watching for changes and the api server says: hey, there's a pod and it doesn't have a node name. The scheduler then says I'm going to assign that pod to a specific node. The api server is saving that state in xcd and then we go to the cubelet who's. Looking for any newly assigned pods, the api server binds the new pod to that node.

A

The cubelet then creates that pod and starts the container with the container runtime. Then the cubelet sends back the pod status to the api server and the api server saves that state into etcd. So that's what happens when I run cubecontrol create-fpod.yml.

A

So why is this time-sequence diagram different and potentially a better learning tool than the block diagram of kubernetes that we saw before so? The first reason I really like time- sequence diagrams- is, from top to bottom. You see the http requests flow and they're representing time. So you see what happens first, what happens? Second, what happens? Third and you get a sense of how this request triggers different operations in the kubernetes cluster. So that's one thing I like another thing I like is there's a clear separation between the actor.

A

That's me the control plane, it's on this light, yellow background with the components of the control plane and then the node, so the individual node in this case is that light blue. So you just get to see the separation of what am I doing. What's the control plane doing and what does a node do?

A

Another thing I like is that the actor icon, which in this case is a stick figure of me. It invites you to think about who and what can communicate with the api server.

A

Sometimes it is individual developers uh pushing code to the api server, maybe for a staging cluster, but often we have really complex ci cd systems that are actually creating the pods, and so the actor could be something automated like jenkins and the time sequence diagram, because it has, those icons invites you to think about who's. Actually doing that thing, that's causing this sequence to happen, and then finally, the thing I like about the time sequence diagram is that when the api server and the cubelet communicate so the first time that happens is in step 8..

A

You can see that they're located on two different virtual machines, there's that white space in between them- and you just get a sense of like oh okay, there's a lot of networking things going on because there's at least two virtual machines talking to each other, so really pretty interesting.

A

So that's one alternative way of visualizing kubernetes is through a time sequence diagram. In that time, sequence diagram we're actually like getting a sense of what events are happening uh to create that pod.

A

So, let's go on to our second way of visualizing kubernetes, which is a combination of distributed traces and diagrams, so distributed tracing is sometimes also called distributed, request tracing and it's a method to profile and monitor applications, especially microservice applications, and the cncf has the open, tracing and open telemetry project that creates standards for how tools that do this are built.

A

I really like distributed tracing tools because of the visuals that you get and I've done a lot of learning by seeing distributed traces in production.

A

So the example that I'm going to show you today is: I have deployed a sample application, it's called meow micro and it's to go micro, microservice, microservices, there's two parts to the application: the meow client and the meow server and the mail client accepts request via a rest api and then uses grpc to communicate with the mail server, and I want to give a big shout out to fernando diaz at git lab who published a tutorial and the meow micro code that I'm using this example.

A

So what's going on, is we have a kubernetes cluster? The kubernetes cluster has an ingress, nginx ingress so that traffic can enter the cluster and then jager, starting at nginx, actually traces the request as it gets to the server, which is incredible.

A

So let's take a look at what's going to happen, so we're imagining that a user sends this curl request to an ip address, the meow endpoint and we're sending a post request, so we're actually sending this data and it's a key value pair. The key is name and the value that you're sending is meow mixer.

A

So this is what it looks like in jager, and I really like this so at the very top of the jaeger user interface. We see we, this request has come in through the meow endpoint and then, if you look below where it says, service and operation, we've got nginx and that endpoint that endpoint and then we have an indentation. We see that request goes from nginx to the meow client and there's a function in the meow client service that sleeps the application.

A

The nice thing about jaeger and a lot of distributed tracing tools is that you can unfold the trace and get a lot more information. And so, in this view, we've unfolded the sleep trace and we can see a lot of information like these tags, the process, the host name. The thing that I really like is that you can see the ip address of the node that this service is running on, and so, if you sort of know the layout of your kubernetes cluster, you can say.

A

Oh it's actually specifically on this node, we'll look at that a little more closely in a moment, so some benefits uh for creating mental models that distributed tracing tools offer the first one is the flame graph, which is what we see this graph, that's moving from top to bottom, and so the flame graph, like the time sequence, diagram, shows the request move from top to bottom, and it represents the passage of time, which is a really important concept when you're trying to debug a humanities cluster.

A

The second thing that I like about the jaeger traces is that it shows how many milliseconds it takes for each request to complete, which is incredible.

A

The next thing I like about the jaeger user interface is that different colors represent different services, and so, in this case, nginx is represented with yellow and the meow client service is represented in that teal color.

A

And finally, I already mentioned this, but just to underline it again, you can unfold each trace and get more information like which virtual machine is that node running on that's hosting that specific container.

A

So if you have people who are new or curious about kubernetes setting something up like this and letting them unfold and explore is going to be a really powerful learning tool, so this is distributed tracing, but in this particular example I think distributed tracing plus a diagram are particularly powerful.

A

So I created this diagram using lucidchart. That shows the request that we just saw the distributed traces for, and so let's take a look at this on the right side of the screen. There's the computer and you see like I would be running that curl request so sending some data to the microservices.

A

So we see first, that request hits the ingress enginex load balancer, which is how traffic gets into the cluster and then that load balancer sends it to the ingress nginx controller, and then the controller sends the request to the meow client. uh That request then keeps going toward the meow server and then the sleep function gets run. So the way that I developed this is that I looked at the distributed traces in the jaeger ui and then built out this diagram on my own.

A

So some benefits of this diagram that you don't necessarily get with just distributed tracing the first. Is you see the computer outside of the cluster, making the http request? Once again, it invites you to wonder well who's, making that request or what's making that request? Is it an actual person or is it some sort of automated system?

A

The second thing that's interesting about this is that you see the request, go to the load balancer and then to the ingress controller and then the server, and so you get a better sense of how does traffic get from the internet into a kubernetes cluster and a load? Balancer is usually the way and then the most interesting part of this, I think, is that the control plane is not involved in this particular process.

A

Somebody sending a request to an ip address with a specific endpoint, and so you can just see, there's there's nothing going on in the top part of this diagram, so that is distributed tracing plus a diagram which I think is a great way to understand the passage of time, but also sort of understand how do things flow through kubernetes um when you're, just interacting with the applications that you've deployed.

A

I also want to call out a very cool project. It's a weave works project called caseband and brian boreham uh created this project, and so this project is very cool, because what it does is it's using jaeger to trace kubernetes events. So in my distributed traces and in this diagram, I'm not tracing kubernetes events, I'm tracking an http request that goes into a cluster, but this particular project shows you all those events that my time sequence diagram was trying to represent so check out this project.

A

There is a cube con talk about it from earlier in the year 2021 and there's a lot of information about it, and it's really exciting. It's still experimental, but something that I'm super excited about.

A

So I just talked about some of the benefits of distributed tracing uh plus diagramming. uh Before that we talked about time, sequence diagrams and now we're getting into like the wackiest like the most fun part, which is, you can represent kubernetes clusters with a 3d model, and I don't mean a 3d model like you're, using some software to develop things that look like they're three-dimensional.

A

I mean something like this, which I've built out, which is like I used clay and marker and twine to represent a kubernetes cluster. So let's see exactly what I'm representing in that in that 3d model.

A

So this is the exact same example that I did for the time sequence: diagram we're imagining that I am the user who is creating a busy box pod by creating the pod through a yaml manifest so running. This command again cube control, create dash f, pod.yaml, and so this is the 3d model of that particular request.

A

It's similar to the time sequence diagram, but doesn't have things like a representation of the passage of time, but you see on the left side of the model is me at a computer, and you see the request flow into the api server, and then you see on the right side. We've got the controller manager. It's also talking with the api server.

A

We see lots of requests uh going from the api server to xcd, and then you eventually see the api server and the cubelet are communicating with one another and then the cubelet only communicates with one request to create the pod, so it's sending it to the container runtime interface.

A

So I understand this is like a little bit wacky. This is very homemade. You can see like the glue- that's stuck on the cardboard, but there are some real benefits to be being able to touch and feel a kubernetes cluster, even if it's a model that you're making up on your own, not actually touching uh hardware, um so some benefits uh of using the 3d model and some observations about what I learned about kubernetes building out this thing, so the first one is in this 3d model.

A

You do get the sense that only the api server is communicating with etcd, that's true, but you see that there's only one place that those strings are going from etcd and it's to the api server.

A

The other thing you really get a sense of with this 3d model is that uh the api server is like servicing a ton of requests, and so uh in building this model out, I was like oh yeah.

A

We, we call the api server like the brains of kubernetes, and I I have experience with api servers being down and clusters being unavailable, but you could see like nothing can happen until lots of things go through the api server first, and so you get that sense here and then the thing about this particular request that I'm modeling is that you see the complexity of what's happening in the control plane, even when you're creating just one pod.

A

A lot of things have to happen for that image to create a container on a vm that we have access to. So that is the 3d model. So I showed you the time sequence diagram. uh I showed you how to use, distributed, tracing and a diagram, and then the the weirdest and fun most fun. One for me anyway, was the the 3d model.

A

So let's do a recap of why all this stuff is important. So, in order to learn in order to interact with complex systems, we all develop mental models that are frameworks in our minds that represent an external reality, and so everyone that interacts with kubernetes has some sort of mental model about it.

A

um There is a group of people, and I am part of them who I call the cloud native generation, and we are people who have started our ops careers purely in the cloud. We don't have any prior experience with all of the technologies and the practices that kubernetes has replaced and our go-to uh way of modeling kubernetes clusters right now, either our block diagrams and although I don't think they are harmful.

A

I think we need more engaging and interesting ways of visualizing kubernetes, so we can help people in the cloud native generation get up and going, and so the takeaways for you. If you are visualizing or diagramming a kubernetes cluster, you first, you want to create diagrams that show the passage of time. Time. Sequence. Diagram is a great way of doing that, as well as using distributed traces that show you from top to bottom. How are things unfolding in the cluster?

A

The second thing is to use distributed tracing technology as a learning tool. Jager isn't the only distributed tracing technology. There are things like lightstep and honeycomb and datadog and others, and use those both as a tool for knowing, what's going on in your cluster and being able to identify errors, but also use it as a learning tool.

A

And the last thing is that you should build 3d models of kubernetes clusters for people who don't have a lot of prior experience in computing, touching something creates a much richer mental model than just imagining an abstract system in your head.

A

So all of this is important because we want to be able to hire, train and retain early career engineers, who don't have a lot of experience in computing.

A

So I used a lot of resources to prepare this talk and I've listed them out here. There are some outstanding examples of kubernetes time sequence diagrams. You can find here and I created my sequence diagram at sequencediagram.org for distributed tracing. I used a yeager and was tracing ingress nginx, so there's some documentation there and then, like. I said I'd like to call out again fernando diaz and his tutorial, setting up distributed tracing in kubernetes and then look into k-span that tool that uses jaeger to visualize kubernetes events.

A

uh There is a kubecon talk from 2021 about it, as well as the github repo with the documentation.

A

Thank you so much for coming to my talk. My name is kim schlesinger. I work at digitalocean and would be happy to chat with you soon.