Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: First Principles of Cloud Native Technology - Ronald Petty, RX-M LLC

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

First Principles of Cloud Native Technology - Ronald Petty, RX-M LLC

Have you ever inherited a system? If you ever feel unclear the system's purpose, it's time to break it down into first principles. With so many Cloud Native tools available it can be overwhelming to decide which tools are most directly solving the problem (or could solve it). We will take a systematic approach to enumerating and grouping core issues and how the related tools solve them. We start with a review of core operating concepts, basic data and network models. From there we contrast OS level solutions to these problems and their challenges. We next enter the Cloud Native layer and see how they leverage OS level concepts to solve these problems. Finally, we compare potential solutions to core issues at different levels (OS or Cloud Native). This talk is focused on how to break down tools into functional areas and contrast them to each other. Leaving this session you will have more techniques to reason about these tools and model solutions.

A

Hello and welcome welcome to the first principles of cloud native technology.

A

My name is ron petty and I work at rxm. Rxm provides instructional consulting and advisory services in the cloud native space. Some of the things we have worked on included creation of the kubernetes certifications, the ckad cka and the cks prior to cloud native technologies. I worked in energy and finance working on high performance computing systems.

A

So why are we here we're here to talk about using first principles to understand cloud-native systems, so to do that we want to set up a scenario, and the scenario is: is you've inherited a system that you did not create more to the point, the system has some aspects of cloud-native computing, so you know using kubernetes in the cloud and we'll kind of leave it at that for now, and so, unless somebody provides you a lot large dearth of information and experience, it can be hard and challenging to understand uh what the system is and what the system does.

A

So here we have the dreaded I've ssh into a machine, and that's all I know right is: is there something that we can do better to more quickly come to an understanding what this system does now? This talk isn't about. Judging, we shouldn't hate the code that came before it's more of understanding, what we have, how to find it, how to confirm it and try to fill in some holes in our knowledge.

A

Now this talk comes out of experience of teaching courses and consulting around this technology, and so uh at this point, we've now taught thousands of people around the world of various skill levels and there's been some pretty common trends in kind of the holes that need to be uh filled. It will, it will vary per person, but the idea with this talk, hopefully in 20 or so minutes, will actually give you some um ideas to think about how you can construct your own plan using first principles to help understand the system that you have inherited.

A

So how might we? You know more tactically, start this process. One way to do this, of course, is to set some goals so as an engineer goals, typically look like fixing bugs right.

A

We want to do something here and now uh could be something a little more involved creation of a feature assuming it's a little more complicated, not always, but but can be very often, we may go beyond that more of a time, commitment right, so maintaining so that's a different level of activity right, that's uh many features and many bugs right so again we're talking about in generality, right, an entire. You know caught a sas type system that you've deployed for your for your business.

A

Then of course, uh once we know what we're doing and we're maintaining it. How do we improve upon that? um So again, we've got this system kind of day, one um we're being a little naive here, we're assuming that there's not much to go on. So what are some things we can kind of do uh to to have a better understanding right. That's really the goal of this talk, so one approach is through first principles.

A

Here we have a bunch of pithy quotes, explaining kind of what first principles are, or at least start to talk about it, but we'll just start at the bottom general reassemble from the ground up and question assumptions. So for us we've inherited the system. We want to know what the components are, um and you know it's how far back? Do you go right in a perfect scenario? We go to some elemental level, but we don't have unlimited time.

A

So we have to balance those kind of two competing uh world views, one of full understanding versus getting to a point where we can start to build back our understanding of our overall system. So the idea here is to kind of lay out some of those areas to look and how they relate, and hopefully you can find the a path that works for you.

A

So a couple terms just so we're on the same page: a system is a collection of things working together for us, it's groups of processes, whether we're building them or using some kind of hosted service. One that's lesser discussed- is the the user so prevailing political or social order. So that's a system too. So there's a kind of a system on top of a system right we've got this sas type deployment and we have our internal users right. Customer care finance the engineers data analytics right all these kinds of efforts.

A

Then you have your end users right. It's not all engineers all the time. So, while we're here filling in our mechanical knowledge, um when we combine the components, we're going to need that uh those other world views, so we're going to see that we're going to want to pick up.

A

uh You know out of our chairs and go talk to these people and see if, if what we think is the world is what they are actually participating in so related to this system, this cloud native system of ill definition that we've inherited uh there's cloud-native development. You know why are we using these tools? So, just as a as a reminder, you know speed the market.

A

um You know kind of that marketing level stuff. That's you know good high level description, uh the red hat one here get a little more meat on it, so um responsive scalable fault, tolerant, apps anywhere in the public, private or hybrid cloud. So one thing to highlight here is very often we inherit these systems. You know make it a little more concrete. We're assuming we're in some kind of you know public cloud right so now for any kind of substantial system pieces of the system are things we wrote.

A

Things are created by others, but we use them like a kubernetes, and then there are services that we may use that are hosted like a hosted database, and so the overall system has all these components. So how might we you know start to understand why they're there uh or even more simply, are you know? What is there? How do we even find that so the reality check happens right so day? One you, don't you know again.

A

The scenario is, we don't have much to go by, so what can we do so if there is some existing documentation? That can be good. So it's kind of this upper left quadrant here and you know it doesn't matter. What the diagrams are. The point here is: is that there's two diagrams right? If you look hard enough, you probably will find more than one piece of documentation and they can have some kind of confliction.

A

Maybe it's a version, one version two and it's nice and easy like that other times, it's two different interpretations of the same thing, so it can be confusing, but we look for them anyway right. It saves us some initial work. Bottom left, uh there's issue tracking, you know, assuming we have a sas with many uh disparate processes, we may have many uh projects and many repositories uh that may be true, but even with all them finding trends which repos are busy, which ones have the most issues. What are the types of issues? You know?

A

What are the issues now versus six months ago right, just spending an hour or two just kind of surfing through these things really can give you some context on. What's going on on the top right is a lesser known, techn uh technique, which I think you know in the current reality of not knowing what's going on, is a very good one is look at a bill even just last month's bill again assuming we're in kind of a public cloud scenario. It'll have an itemized list of all the services we're using.

A

It may not give us all the data flows. It may not tell us every process, that's running on a vm, but it does. Let us know it even exists now that doesn't mean we're actually using it right. You know worst case this bill encompasses things that have nothing to do with our sas best case. It's our walled off production, environment and it these are things that are in it and nothing more right, but that doesn't mean we're actually using them all right now.

A

This isn't critiquing, you know, is it good or bad, which it can highlight? You know maybe a real expensive database or unnecessary scaling, but that's not the point. You know we're kind of in the scenario we don't know what all this is about. So how can we kind of get a good quick view? A bill is a very good way um to do that. Now, of course, you may have services in another cloud provider and you don't even know to ask for the bill right.

A

So don't forget that kind of next step you may have known the main part was in amazon, but maybe you have parts in google right talk to finance right talk to a manager who had been involved right just see. If there's other parts right, we can get there's other ways to look at it. We can see network connections and things like that, but at a very high level, very easy, powerful way to do it and then the bottom right is that shared context.

A

So we you know again we're we're assuming we're relatively new to cloud native technology. um Even the projects in the cloud native space are continually developing. So we even though we're using particular technologies. um There may be better.

A

You know white papers and scenarios with a related project than the project we're actually using so once we know we're using something we can go, look for its document documentation, but that may not be enough, so maybe look at something in the same space to see how people are using it again, not the critique not to replace the idea is to get an understanding of what the power of these things are again. Assuming we're newer to this technology.

A

So with this idea in mind, I've talked to some engineering managers with some very large scale systems. You may know, including twitter, uh some of the original managers and and other projects, and you know after interviewing people, I wanted to get their kind of ad hoc, because I wanted that you know experiential like what would you want to do on day, one you know without a ton of like preparation uh to deal with this scenario.

A

You know you've inherited a system, a cloud native type systems more specifically, but you know they start off with just general systems. So the three questions that were asked were: how do you go about understanding the system you inherited?

A

How do you know if it's doing what it's supposed to be doing and how do you share that knowledge with others? So we'll go through a couple of of these managers feedback, so here's the first one so uh first goal find out. You know what it's you know, what it's doing, what you just inherited is find out what it's supposed to be doing talk to users. You know how do they use it? That's the very real world right. It's what they're doing right now, you'll learn about web portals.

A

That may not be documented right, some kind of web interface, or maybe some kind of reporting engine mechanism behind right, we'll start to see the entry points pretty quick. There is the dream of what they want and what frustrates them, but you know don't do that on day. One right! That's that's further down the road once you know what we got then, of course there's the verification of these things.

A

So um at a lower level, if we have the related documentation for these components at an api level, communication, that's great schemas on the database side tend to be a little easier. They have tools to show the schema to generate the schema right. Apis have them as well, but they're not as often deployed so a combination of those things can give us a pretty decent view on how the conversation goes, not necessarily who calls who, but what the it may look like. Then, of course, the tool selection matters. If we see particular tools being used.

A

You know if there's a queuing system in there, whether it's hosted or not, um we we can make some assumptions like okay, there must be a need for a buffer or a retry queue right, so we can kind of know. Okay, maybe that's part of the system that performance really matters right, or maybe it doesn't matter. Maybe we cue it because we we slowly work over something right, but it gets us thinking in those those terms all right now, next question: is it doing what it's supposed to be doing um mentions here?

A

Acceptance types, so this is kind of the end user experience. Does you know the features I care about? Is it doing that again? There's in the testing world there's many terms. You know we can slice and dice this, but if you're frantically learning a new system having these acceptance tests is a great way and if they don't exist, create them now acceptance tests tend to focus more on, like actual features that we want. uh A related idea is a category categorization test and that type of test is with this input.

A

I get this output, I don't care if it's good or bad, that's just what it is right, so it's good to consider doing a categorization test as well. That's a technique used very often in legacy systems, because no one remembers how it was built right. So in the cloud native world, if this is new to you, it doesn't hurt to go look at what was done in the past with legacy systems. Those techniques can also kind of help. You learn these things, so it's another way to learn.

A

Then. Finally, um the you know the measuring right so measure. You know that's another one. Is it doing what it's doing so it's that ongoing metrics right. We can't tell what a snapshot is it right or wrong, but we can look for consistency and behavior what metrics and the bottom question? How do you empower others? It's the sharing of that information right, don't don't hide it. Engineers tend to to measure and not necessarily share it's not nefarious. It's just very often happens.

A

That way do consider sharing it, because that also gives other eyes and other uh conversations right next engineer uh slightly different view, but some overlap. So how do you learn a system you just inherited uh here? It's a data flow analysis. Again the protocols and messaging we're gonna, see, there's tools that help us with this. Then second graph, graphing, that communication flow so seeing the relations right so again, we'll see there's tools later that help us with this uh next question: how do you know if the system is solving?

A

What it's supposed to be doing here, metrics matter once more, we want to see those trends. A snapshot is, is too too, uh you know point in time. We need to see trends. Ultimately, we need to see those acceptance tests to prove it did what it did. But in you know it's a system and presumably a lot of activity. We want to start with the generalities and then start to dive into the details again kind of the principled first principles, approach, okay and then um the bottom here. How do you empower others again?

A

It's back to sharing, so you tell people what you see, you listen to what they say and you try to, as they say, bring to light the parts that are missing.

A

So how do we go about understanding? um Really? What we want to just say here is you inherited the system, so you need to do some reflection. What about these technologies that you do see on the bill via talking? Do you know, and what do you not know right that gives us another opportunity to quickly figure out what we need to do, whether it's we're learning it or finding someone? Who knows that thing? We don't know right. We try to bring them in and fill in our holes and our knowledge.

A

So again that requires self-reflection very often, if you're, just fighting bugs and doing it, you're you're, very myopic, and just doing that again, you don't have a bigger plan. The idea is, we've inherited the system and we're going to maintain it for the long haul. So how do we? How can we start to do that?

A

Another thing kind of echoed earlier reflect against others share the information see if, when they tell it back to you, do you come to the same conclusions or not. You know what does that mean so think you know think about it as a group, not just engineers customer care right, uh other managers finance right, other groups. They will have their own tools and insights into these systems.

A

Okay, now there's a continuum here from a technology point of view, so it's less esoteric and more into the tactical stuff. This continuum on the right hand, side cloud is generically all the larger encompassing technology.

A

Kubernetes is our orchestrator right running these pots. Pods are a collection of containers, are one or more processes, so there's technologies at each of these levels and for us in cloud native land. These you know this is kind of the spectrum and so starting on the right. How do we start to drill through these we've mentioned some concepts like accessing bills, accessing the hosted dashboards right for the vm, maybe maybe again those initial scenarios. Maybe we deployed kubernetes well, it's on a vm that vm probably has performance metrics right.

A

It won't break it down necessarily to any particular process, but it gives us at least something to start with um same thing with logs. If there's ability to do log aggregation, take it take advantage of it if the system already exists, there's probably I am type controls access controls. We can you know without knowing all the security settings, we can kind of see what is used to access and how many accounts there are, how often they change we start to get a feel for for how these things are all put together.

A

If you kind of slide down past kubernetes, we get into the pods in the in lower level into the individual machines right so very often in teaching of this technology. This is generally where people tend to be the weakest. It's actually ironically distributed. Computing is a big, complicated subject, but it's the one that people can easily talk about because of you know horizontally scaling and things like that and collapsing of data right, but knowing the individual, lower level technologies is one of those harder areas, and so here you know, can we start to gain knowledge?

A

Well, if we can run a pod in in the production cluster, we can run it locally if it fails because of dependencies. That tells us something right. We don't necessarily have to have known there was dependencies. We can just experience it right, but you know a lot of people don't consider like just running the thing locally right. It's it can do that um same thing with just an image right. If there's multiple images in the pod, maybe we can run individual images right and then working back.

A

We go all the way down to the kind of process level in the related technologies for sharing resources through name spaces and the security mechanisms. Those things provide just a brief example here, we're running a um uh you know, a pod here with nginx and we tail the log that's at a higher level, kubernetes level.

A

Then we jump all the way to the left right at the process level. We get. We get the the pit here. What can we do with it? Well by knowing some of those left-handed? You know continuum concepts here, we're showing that we can echo output into the output stream of your pod right, your in this case, the nginx container in that pod, and so again, when we go to tail, we see our message hyzer now. This isn't meant to be a co.

A

You know it is a cool example, but it's not meant to be a compelling example like this is something you should do. It's just meant to show like there's high level concepts and then there's low level concepts and they can meet as long as we know where to look, and so that's the goals we want to fill in.

A

So what are the tools that are some tools that are available to do this again on the right hand, side kind of cloud? We don't talk much about that because that's you know whatever the hosting provider provides, but there are things there to learn in the middle kubernetes side. Let's not forget. Kubernetes itself has a ability to produce logs and metrics and then there's things we can deploy to fill in some of those gaps. So say we have, you know basic metrics, but we don't have any way to analyze them or do anything fancy.

A

That's where something like a prometheus comes in right to aggregate all of our stats of our software, but that takes work right. We may have to open up ports and deploy code, there's some effort there. So, let's not forget, there's some built-in tools as well and then, of course, we can go down to the communication level right between things right. That's where we have those service mesh tools like an istio or link or d.

A

On the left hand. Side are things we don't necessarily have to install, and this is why we like to highlight them. So at the cli level, the terminal level, the shell level um there's just things we need to know. We need to know like what does return codes mean um inside of a shell context, um same thing at a process level, what is exact and forking at a high level? Do what does signals do right? Those are the methods of control.

A

How do we isolate and share resources right? All those things then, of course, there's tools to manage that right, the cri you know docker and company, um there's the cni plug-in to help at the network level and and the combination of these things. So what are their core technologies? How do they work? How do they log right? Just because you know how to do kubernetes log, don't forget, we can potentially go on the node and look at the individual machine to help debug issues and understand things. Then, of course, there's the nodes themselves.

A

We have a system d uh as an inits process system that comes into play in a lot of cases. uh Falco systig, uh you know any kernel level system call monitoring, that's a lower level tool that applies to any process, and so now we can see what's going on, I may not have any idea what your code does, but I can start inspecting it um as it's actuating to see what it does. This isn't a comment on security. It's a comment more on just understanding what it's doing um good or bad.

A

So there's some questions here. What's easiest, what's the most powerful? What is the easiest to share? In my opinion, based on what I've seen in general is the left-hand side by knowing these lower level tools? We start to you know again: that's the first principles, parts break down these tools back to you know the components they're built on right, docker, some built on name spaces and resource control. Kubernetes is scheduling this across multiple machines.

A

The cloud is moving even beyond that, and so it's the combination of all these things, but I think, from a tactical point of view back to the self-reflection. What on the left is is missing. That usually will be one of the areas that will help you the most. If you can start to fill it.

A

In are there other things? We've inherited a system, there's pipelines, that's another great view of the world and it's probably reality right. If you push code and shows up in production, it is what it is so studying those steps is a great way to start to see some critical features, but remember it's not everything right. Just because you had a pipeline doesn't mean it was all everything was automated right. There could be systems um that are not so try to fill in those gaps right again, you're not going to do this overnight.

A

So you want to you know, take a very systematic approach at what you see and verify it right. That's that's! Really the kind of game we're playing here beyond that. There's security. There's optimization right! There's all these things! That's you know the goal here. Isn't um you know trying to give you advice on how to optimize things the data you'll get from all this will help you do these other things be more secure, be more optimal, get get to those improvement phases right, not just maintenance phases, but even getting to the maintenance mode.

A

If you're new to the system is difficult and in the cloud native world, we have lots of tools to fill in these holes and metrics and and controls, uh but without kind of knowing what their pieces are individually. uh That's a step, a lot of people miss they. They hear the high level terms and they just want to go for it, um but then they hand it off to someone- and you know they can kind of leave a mess.

A

So what we're recommending is take a risk, adverse approach here and build up a plan study these things over time and they will continue to pay off uh in the long run and be sure to share this information right. You don't need to say everything to everyone, but uh get that reflection against other people.

A

Okay, uh for for our more junior friends out there, here's some um more specific things you can can look at and for the more senior who may have forgotten some of the the lower level technologies which um can happen um again on that left-hand side. Here's some things to start uh considering. Most of this is actually very tactical, like processes in user space in the proc file system, but some are more modeling right. So osi model is a good one. To reflect against, doesn't mean we do it, but we reflect against it microservices.

A

You know: why are we having these hosting platforms and containers and why is it stylized this way? You know it's easy to say in a conversation, but you know again take that next step and try to do a little bit of self-study to make sure you fill in the blank some of those tools we mentioned prometheus service meshes things like that highlighted them here again take advantage of that landscape right go see what's out there. A lot of them have super great documentation.

A

Again we only have 20 minutes or so 30 minutes here. So we we. uh We can't cover everything. So there are a lot of good stuff out there. So please take that next step and do some fundamental research and again take advantage of first principles here, a lot of cool technology and there's a lot to learn and it's a lot of fun all right. Thank you for attending um a talk. I hope you have a great conference if you have any questions about any of these things, of course, teaching and consulting.

A

But even if you just have a question, a technical question, um you know a design question. If I can help you, I will so please don't hesitate to contact me have a good day.