Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 20 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Guide to the Galaxy (E48): OpsCruise: Observability in Cloud Native Systems

Description

Join GitOps all-star Hilliary Lipsig and guests for a fun and engaging trip through the galaxy of GitOps. She’ll take us from orbit down to the street level of how we can achieve and manage our infrastructure using modern best practices.

In Today’s Episode Hilliary is joined by Cesar Quintana from OpsCruise to discuss observability!

Want to learn more? www.OpsCruise.com

Try Red Hat OpenShift: https://red.ht/3sMaW7O

Follow us:
Hilliary Lipsig https://twitter.com/Int2Caffeinate
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1

What is GitOps Guide to the Galaxy?
Every other Thursday at 3pm ET host Hilliary Lipsig and guests dig in to discuss everything in the GitOps universe, from end-to-end CICD pipelines to creating Git workflows.
Learn GitOps https://red.ht/3saqlzS
What is GitOps? https://www.redhat.com/en/topics/devops/what-is-gitops?sc_cid=7013a000002qG1BAAU

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3LGCIeD
OpenShift on GitHub https://github.com/openshift/

#RedHat #GitOps #Kubernetes

A

A

It lied again good afternoon, gadopsians, um sorry that we're starting a couple minutes late. We had a actually, we didn't have technical difficulties, I'm just a chatterbox, and that's really what happened there? Oh I'm, not even going to lie um quick couple of announcements again before I introduce our guests, um so uh this one's big, this one's very big I'm, extremely excited to announce the return of Christian Hernandez to the get UPS Guide to the Galaxy stream.

A

uh He will be rejoining us back um next month, after um so he's at kubecon, North, America and Detroit right now, specifically for like um or about to be rather specifically for good Ops con and, of course, the greater convention at large. So after he's, you know done with traveling. He has returned to us here at Red Hat um after a brief uh brief time in the great unknown and uh so yeah.

A

We're very, very excited I'm, very excited to announce that he's going to be rejoining the stream, as my co-host um later uh next month. um On that exact same thing, we're going to be figuring out the rest of our show schedule. I had something set somewhat in stone, um obviously, with his return.

A

That becomes more flexible, I believe our next stream is still going to be um Argo rollouts on November, 7th um and then actually later in November I will be away on vacation, so I don't think we're going to have a second stream in November and then we'll get back to you all. First week of December, slash top of the new year with, like everything that we're planning to do going forward, uh the dynamic duo is going to be back together, I'm extremely excited.

A

um So that said, it's my turn to badly introduce. As per usual, today's stream guest um I didn't actually practice saying your last name ahead of time. Yeah.

B

Quintana yep.

A

um I'm not going to even try to explain your job I'm gonna, let you just take it from here. Yeah.

B

Absolutely no thanks Hillary, so yeah! um Yes, my name is Cesar Quintana I am a principal Solutions architect at Ops, Cruise. um Basically, sales engineer right, so my job is to interface with customers and uh make sure that we uh get them properly set up on our awesome. Observability platform Ops and uh you know, I've been in the I.T space for quite a long time. uh Maybe I guess 16 years at this point, something like that um and uh and in the observability space for probably about the last nine or ten.

B

um You know very. They come from a background of a lot of APM platforms. uh You know major APM platforms, you think of them yep those are the ones and uh and so uh yeah. So it's it's awesome to be here and I'm very happy that I get to join. You Hillary.

A

So um let's just go ahead and ask the stupid question: 8 PM, that's one of our least favorite, most favorite things, a three-letter acronym or TLA. Can you expand your acronym.

B

Absolutely so APM application performance, monitoring, writer, application, Performance, Management I, think it's gone through a few different iterations, but I think at the core of it. Apm uh really refers to code tracing right, the distributor what's now known as distributed tracing the instrumentation of uh of of code libraries in order to track and identify um request, basically conversations the flows of your business operations.

B

um You know some platforms, call them business transactions, we call them Trace paths, uh you know you. The point is those are business critical. You know, operations in your um in your application and yeah. The the whole APM construct is built around instrumenting those those pieces of code and really following a request, a transaction from the beginning of its life cycle all the way through the end and understanding the performance, especially now with microservices right. It's it's super super key.

B

You know before it used to be just one giant, your giant monolith and everything was internal to it. But now, with uh with the increase in in uh in distribution of the services, the you know, the whole microservice Paradigm um being able to track uh at each hop what uh what pieces of code are being run. What functions are being run and being able to trace their uh performance as well as any problems in their run? Is super key, so yeah yeah.

A

Absolutely 100 yeah um distributed tracing is a big topic. It comes up right, I'm, a reliability engineer. It's comes up, I, almost every single day, I guess it's it's one of those things where, um especially especially during an incident where it's not immediately clear. What's going on right um and you know back when you talk about monoliths right, uh how many times I went to look at SSL access logs to kind of figure out? What were the connections were coming from? That might be, you know, part of an issue. I can't count in today's Services.

A

It's not that's not really the same way anymore right. You don't necessarily have that one one Ingress point exactly in the same way that that tells all, because there's a lot of little bits that I'll talk to each other in in different ways and I. I know that you know this and I know this.

A

This is not my first time uh seeing Ops cruise and my favorite thing was the network tracing visualization that you had in there so like, if you weren't planning to show that I hope you do now, because that's your favorite part, um but I do know that you have some some stuff to show to the audience a live demo to go through.

A

um You send some uh some links to us earlier as we go through this as we let you go through this, we'll kind of put them in the chat for people to take a look um and I'll. Just let you like, let's just jump into it, get started. Go ahead. You'll have to I, actually showed you this before we started. You'll have to share your screen and then I'll enable your screen, and then it will come up beautifully. So.

B

A

There's a delay in the share: it's just uh it's waiting on Johnny or I to run the production aspect of that.

B

All right that sounds good, so uh Wally sure. Let me actually move you guys, so I can still see your guys's face when you talk to me, but.

A

You don't need to look at my face. That's unimportant! um No! It's terrible right! I! Have this giant! It's a 49 inch monitor. It's mounted on the wall in front of my desk right.

B

A

My camera sits on top of it it's right here. This is where you guys are right there.

A

um So if I'm looking here, it's great you've got me dead on right, but if I have anything else right um I'm over here, you guys are getting the side of my face and that happens I just in in in 2D world that we live in with these, with these uh remote streams and so forth, I think we have to just get look, get kind of used to looking at the side of each other. Every now and again,.

B

Yeah yeah, it looks like you're distracted once like you're like you're, watching something else right when you're at reality, just paying a lot of attention. Yeah.

A

B

So yeah, so hopefully you guys can see my my screen here.

A

um You can make it I I, don't know why I pantomime this like this is helpful, but like magnify it, that would be a better pantomime, but.

B

No I mean I can zoom in more. uh Does that help yes, yeah yeah, okay, so I will zoom out and I'll zoom in again just to show you kind of what we're looking at, but this just to give more of a bird's eye view.

B

So what you see here is um is: is our up screws landing page, so you know what just just to set the the tone for what we're even talking about, because you did mention APM at the beginning of the call, but I I want to go a little bit further than that and say we're not just APM, right and I. Think really the industry as a whole is starting to move away from just calling themselves APM, because I think APM is just a snippet of really what the larger observability space has turned into right.

B

So very, very key here. The reason I bring that up is because this view that you're seeing here what it, what it is, is you know I'm this. This demo happens to be inside of AWS. This is a mostly kubernetes cluster you'll see some things like virtual machine processes scattered throughout.

B

Maybe some past Services you'll see an RDS instance here you see a oops you'll, see a um a load balancer here as well, so you'll see some of those, but um but but most of this is kubernetes and the reason I uh what I'm, what I'm trying to get at is that.

B

In fact, this view is built without actual code instrumentation, um so there's that whole industry of moving from just APM or just you know, standard uh like node, monitoring or VM monitoring or just a specific entities metrics, and that's it or a specifically into these logs and that's it. The whole industry is kind of moving to somewhere a little bit deeper, and so specifically, when it comes to APM, we love APM. We love coach racing, we support code tracing but funny enough.

B

This view is built without any code tracing and that's some of the marbles of the new um ways that open source- and just you know the new observability paradigms have really allowed us to be flexible. So I'll show you guys what our code tracing looks like in a sec but really really cool. This is this is done all leveraging evpf. This is all leveraging um contract as well, and so what you're seeing is a basically a real-time view of these Services interacting with each other and I.

B

Think Hillary correct me if I'm wrong, but this is what the the map you were referring to earlier right.

A

The thing this is the thing that I was like: I love, this yeah.

B

Yeah, so this is this is one of our biggest. uh You know, features that customers love and the the really cool thing about this. As I mentioned, you deploy Ops group, so obstacles is um really. What we're doing is we're leveraging the underlying um open source tools so actually I'll I'll switch to my architecture. Really really quick and I'll come back to this. When we talk in more detail about it, but um you know here on the left side, you have your your components that are being collected could be pad. Services could be kubernetes data.

B

uh You know it could be messaging layers, whether they're paths or on-prem. You know virtual machines, containers Etc right, so you grab data from there. um The type of data again metrics events, traces logs, configuration, Network flows, but the way we grab it is really leveraging the open source components, things that are now becoming pretty standard inside of modern application environments. So things like low-key for logs uh Prometheus for metrics, Jaeger and tempo for traces. We support the open, Telemetry standard.

B

We take data from cloud, um so this is really where we're bringing all this data. Excuse me so that includes the evpf component that allows us to do a discovery of of all these entities and uh and yeah. So this is one of the really cool things now, once you deploy that stack, which, by the way, if you don't have it it's okay, we you know we can deploy and, uh and- and you know about three four minutes- just a Helm, a couple of helmands you're up and running, and this whole thing.

B

The cool part, is that this whole thing gets discovered in you know about a minute or two after you deploy. You have this full map. It doesn't matter what language you're using it does you don't have to install any specific language agents? You don't have to do anything special because of the underlying way that we can now collect data. This is self um discovering and constantly updating. As soon as you add, new workloads yeah.

B

So um so the other thing I want to call out is I mean once you're in here. The really cool thing is that not only are we showing you I mean if you look underneath the covers. What we're doing is that we're providing um we're providing uh topology right.

B

Your applications, apology now dependencies and you can know the relationships of what's talking what, if I zoom in a little bit more as Hillary, asked me to early uh earlier, um you can see that there's arrows and uh and kind of graphical indicators in which direction traffic is Flowing, so you can see what's calling uh where and in fact not only are we doing like network discovery, you know of services talking to each other, but what you're seeing here is a mix of uh of uh endpoint to endpoint connections and also configuration data metrics Etc, and let me show you what I mean by that.

B

So by the way, um I'm going to turn off, there's some components like the the components that send the open, Telemetry data or the open source data out into Ops cruises. Some of these are running inside of this cluster, so I'm just going to turn those off, and so this is actually really handy for when there are teams that only want to see um you know their specific name, spaces or specific applications. You can filter here by labels.

B

You can filter by specific pods I'm just going to turn off the Obscure stuff, just so that so I can tell you what we're looking at obstacles and collectors. So we have a couple of apps in here and the one that most of you will probably be familiar with.

B

Is the Google microservices Boutique right so we're running the online boutique sample application here, and so that's what you'll see if you're, if you're used to um sorry, if you're used to uh that application and you've seen it or run it inside of your environments before these service names will be familiar, shipping service card service front, end, Etc um and again all this was Auto discovered, um but yeah I turned off some of the stuff. Just because this is what's important to me right now, and in fact I can save.

B

You know different views like if I, if I want to see everything I can just apply my all services view versus if I'm, only interested in like I mentioned earlier, my online boutique, where I can apply those filters and I'm only seeing you know those specific services and I can you know tweak, can share Etc but I'm just going to clear everything back out just turn off pop screws and yeah. So so all this data is being collected. What I mentioned is that we're also grabbing the Telemetry data and it's being represented here.

B

um So if I click on any one of these entities right, I, just double clicked on a pod, I'll zoom in a bit I just double clicked on a pod and if I click on that pod again I. Can you know you know, there's there's just the container. That's uh that's also sitting inside of that pod is visible there, but I clicked on the Pod itself and I can see the checkout service right.

B

So this is this: is a checkout service, pod I'm in that context, I'm looking at things like uh configuration data right, so the labels that are attached to this pod, which are actually something we can use to group and I'll, show you that in a sec labels attached to it configuration data things like the host name, that this pod is running on, whether it's an Ingress pod or not the namespace, the AP address of the Pod, the host IP address when it was created.

B

So it looks like today at 3 30 in the morning or actually yesterday, at 3 30 in the morning, and on top of that uh you know really important to to context, especially um for like openshift and kubernetes clusters. You know going in and looking at what's running, you have a couple different options. If you have good get Ops practices, um what you what you can do is you can look at your repository and make sure that you know you can see what's the running config for a specific entity.

B

But that means that when you are in your you know, observability platform and you're trying to troubleshoot, then you would have to go inside your other tool inside your repository and you know, check the running config. Your other option is going into the command line and you know running OC, you know get pods or you know, qctl get pods and and looking for you know the proper output Etc. What we've done is we've brought all this data into context, and so now you can just when you're.

B

Looking at a pod and say, okay I see that this checkout service is talking to product catalog service.

B

um You know what what's my config on there I can just click on detailed View and now I'm, looking at the full-on yaml Manifest for that particular entity right so I can, instead of having to go to command line or even go to my repo, which is awesome.

B

You know using git Ops is is ideal, um but having to go back into a repo into a separate place is not ideal when you're trying to especially like you, have a project action outage, and you need to make sure that your Port is Expo, that your container is exposed on the appropriate Port. You don't want to go back into your repo login and then just you know, look for. We've got it all in context. Immediately.

A

um Yeah the other right tool at the right time for the right job type of things. Exactly.

B

Yeah I know exactly um you don't want to be. That's that's a lot of the you know, the philosophy that we kind of are hoping to bring to the industry. It's you know context. Context is King, because if you need to look for information right one day, you might be tired.

B

One day you might have only slept a couple of hours and you have to remember the name of something and then you go to your other tool by the time you log in especially now with like multi-factor authentication by the time you log into your other screen. What was the name of that service and then you're going back and back and it's just it's it's it's juggling it's. um You know a lot of swivel chair operations, it's it's painful! Sometimes so yeah you'll see that a lot of what we do is bring context context context.

B

In that vein, you can see on the right side um in the context of this pod, we're bringing in the Telemetry data right. So things like metrics if I want to look at the metrics for this particular service, I, just click on the metrics link and immediately- and you know anything- that's zeroed out- will just collapse.

B

I could I could expand this if I wanted to, but it's just zero down um so memory utilization, CPU utilization file system, reads and writes Network reads and writes Etc all this data is, you know, immediately accessible with a single click, and this is important right. So I'm kind of laying the foundation of of some of our more advanced features like automated anomaly. Detection, automated root cause analysis, but having all this data is kind of the foundation for what we're going to show you in the context of an anomaly all right.

B

So I'll I'll come back to this, but it's important to know that we have access to all this data. Any events right any events last 24 hours, maybe I'll, go back to the last couple of days. So, as I mentioned earlier, uh you saw that the Pod was created at 3. 31 am. This is how we discover that we grab all the events that are related to to an entity and we see that it was deployed. You know yesterday, at 3, 31 am um and then logs same thing.

B

If I want to look at logs for this particular entity, I don't need to write a complex query. I don't need to. You, know copy the name from my command line and then look at logs for looking at yesterday. Etc I, just click logs and boom I'm there and I can see I can search or if I need to search for anything specific and there's some pretty cool things like we. Just we just released some some interesting functionality around being able to create alerting on on specific logs. So that's there.

B

You can write these custom queries to alert you on online, but you know context is brought to you automatically, even without it um one of the other things. That's really cool is yeah. This map is, is cool um and it's kind of easy, but once it gets bigger, you know we have customers with thousands of thousands and thousands of PODS.

B

You know visualization does get a little bit harder to to parse, um as you have you know, 50 Services you're talking to so one of the really cool things that we've done is you can see this whole network flow, this Services service flow, but in a tabular review, I, just click connections and now I can see the source. I can see that checkout services talking to a few different containers on this IP address I, can see it talking to the core DNS service.

B

I can see it talking to Jaeger because it is outputting, some traces which I'm going to show um but yeah, but yeah. This is this is uh this is one of my my favorite features so understand the the entire list of what it's talking to across different ports.

A

Probably unpopular.

A

Populate that over, like shiny uis with all the at like um and I know, I mean I, know I'm in the minority here right like oh, when you in this is I, want things in list form always and forever at every juncture, I don't care if it's like jira and I'm managing my work day to day or if it's something like this I'm I'm, the worst people ask me what I want a UI I'm like I want a list.

A

You could just give me a list, that's great! That's all I want from a UI ever I. Think every UI researcher, who's ever interviewed me has left the room shaking their head and wondering what just happened. No.

B

I think I think you have a very, very uh valid point right, so so I like our visualization and in fact, I really like that I can go into dark mode. The joke I always make, which is not really a joke, is that you know we all, especially now with people being remote, Etc, sometimes I'm in bed and I'm. Working right, I need to check something and I need to maybe look at an environment I really like that.

B

We can go into dark mode, because that way, my wife doesn't kick me out of bed or make me turn down all the brightness on my uh on on my screen, but so this. So this is nice that we can go into night mode, um but uh what I was going to mention is yeah. These visualizations are cool. Looking at night mode with the blackout with the blue on black is pretty cool, but when you need to ingest a lot of data, you know quickly and really understand.

B

What's what it's you know, we can discover the ports that it's talking over by hovering over like if I zoom in here and I am and if I see this this edge here. I can click on it and I can see the port that it's talking over and you can even see like level four traffic but um but being able to just see in sorry, it's gonna zoom out a bit being able to see in list form without having to go okay. What's here, okay, what's here, okay, what's here, it's it's easily digestible!

B

You consume, find what you're looking for on the spot, so I I agree with you. It's kind of uh going back to what you said: the right tool for the right job, um so yeah, so.

A

I, like dark mode I, don't think you guys showed me dark mode. The last time we spoke I actually won. I find that visually easier to parse, which I think most people do.

A

It's kind of it's kind of where that dark mode isn't the default across a lot of systems, yeah and then uh yeah that part about being somewhere where bright light would be very disturbing to the people around you, um so I have I, have young children, and actually one of my one of my sons is homesick from school today, but yesterday, because he was sick, he was having trouble falling asleep, so I'm sitting there like holding the child, so he'll go to sleep and like also trying to finish up something on my on my phone for work that I didn't get to earlier in the day.

A

So I think that that was that yeah I super relatable I think we all feel it. um We do have a a question from from the audience um so Johnny asks. uh Can you export that uh table data that you were just showing.

B

So a couple of things not currently we've gotten two different requests: one really old school and one obviously more new school um one is: we've got a request to be able to export this data to Excel, and uh it's interesting that that some people still use that format.

B

um I think it's painful, but you know there's different processes, different uh different uh ways of ingesting data at at uh at different companies and you're right you're, like security teams, a lot of security tools take, you know, you know CSV files or Excel files and you know, consume those. um The other requests that we've gotten is for an API, so I know that the API is an active development for being able to export this type of data. You know just want to play against obstacles and say you know from this time to this time.

B

uh uh You know what we're what were all the entities that were interfacing with my checkout service right. You know, so that's something that's being worked on, it's not in the product yet, but it is absolutely absolutely a roadmap item and if it's, uh if it's something you're, you know quite interested in, if you say hey, the office was really cool and you know I'd like this particular functionality, because I can export to security or I can export to whatever.

B

um Definitely please reach out and, and we can talk and and see what your specific needs are and put it in there.

A

Okay, I think we can let you move on to the next topic now.

B

A

Out enough on this, throw something off: that's cool, yeah,.

B

Absolutely okay! So, oh you know what I do want to show one cool thing, though: um when you have all this all this, you know, I just cleared all my filters and now I just have everything in here well I mentioned: there's we have some grouping functionality that can be used on auto-discovered labels or namespaces. So this is like really handy like when you have a bunch of especially like one of the things that I see in um you know.

B

Openshift is Beefy right, there's a lot of different components that are that are running inside of there. This is not openshift I wish I had an openshift environment to show you, but it's one of the things we can do is group right, so um will often will find in like customer environments, when they're running like big either thousands and thousands of containers or they're running something like openshift, and they want to see all the openshift components separated um you can Group by things like namespace, so I can just turn on this filter.

B

This grouping function just say Group by all namespaces and now what this is going to do is this is going to show me um the individual uh workloads that are inside of these namespaces right. So online boutique collectors, Etc and I can, just you know, kind of play with the format here. The other thing you can do is if I go into online boutique based on the discovered labels.

B

All right and again, you guys don't have to do any configuration or all this is just out of the box functionality. um So we have a label discovered called app, so I can just say, uh label app and you just show me all of them and now, instead of grouping by namespace we're grouping by label right. So there's these apps called shipping service, front-end checkout service- you see promptail in here logi. So that's another kind of pretty cool thing that you can do to see. What are all my apps that are running inside of here?

B

So that's uh yeah,.

A

Speaking of openshift, um so actually when, when we when we last spoke, you actually did show me this on openshift, yes, um and so now, I'm going to ask you the thing that I was going to avoid asking you, but we're gonna, ask it anyway: operators have you all built in support for operators and their operands, yet.

B

So no that's still in development. So one of the things that I think we showed you last time is: we have Auto. Detection of you know any at the end of the day. Operators are just pods and containers that are running inside of um the cluster right, so um we do pick up that data. We do see them and just like, we provide uh visibility into the logs and metrics Etc. um All those pieces are automatically you know picked up and brought in um so the operator functionality is visible and trackable through logs Etc.

B

Now there are more specific pieces of functionality inside of different operators that we are working on bringing into the product, but that's not fully fledged out yet just to be clear on that yeah.

A

I mean we only just talked about it like what in the summer, something like that yeah. You know that makes sense, but yeah I think when we talked about openshift.

A

um You know there's openshift as as you've seen it, which is where you deploy yourself and then there's openshift, dedicated right and openshift dedicated. Is it's managed openshift, so you've got SRE behind it?

A

um It's you know, there's there's a few different flavors of that there's, the classic OSD, um which is like runs on AWS or runs in Google Cloud, um there's Rosa, which is a way more integrated version of openshift dedicated with the AWS experience, and then there's um we call it Arrow, which is uh Azure, Red, Hat open shift. So it's uh our openshift Microsoft joint offering um and if you think, openshift is Beefy managed openshift.

A

Is that again um because all of the things that make it managed and make it nice um and and provide that that experience are also are also shipping in it. So it's that's a very it's like they're too. It's openshift is openshift at the end of the day, there's not like differences in opinions really between the two of them, um but it is. um It is one of those things where it's there's a lot of.

A

There are a lot of things too, open shift that make it open shift right, I think we often at least the engineers often call openshift opinionated, and those opinions are often represented as, like you could see them as group workloads on a on a diagram like this right.

A

B

A

For more information on the opinions of openshift, I, highly recommend watching ask an openshift admin and just throw Johnny and Andrew to the bus and let them take all of your questions.

B

You know so I will say that yeah I mean we've seen that in fact, some of our customers, as you are using you know, Arrow some of our customers are using openshift, on-prem and so I mean full support. um Full support is, uh is there for, for all those? um Basically any any. You know. Downstream kubernetes variant is absolutely supported um in our platform, uh but yeah. No there's, there's ongoing work on you know getting deeper into these operators.

B

I'll say that without going into too much detail, because at that point, I'd have to bring in my platform architect to uh talk a little bit more about that, but uh but yeah. So the answer is that it is also being worked on. Thank you.

A

Wonderful love it: let's go on cool.

B

All right, so we have a couple of other kind of cool views that we have in here and then I'll kind of show you um traces and anomalies towards uh you know a little bit more towards the end, but here's a kind of another cool, View and again to we, the the dark mode, is supported throughout the UI. In this case, we are running a five node cluster and you can see all the individual nodes, these large boxes, so now it's more of a physical layout. So you know before it was more of a topological.

B

What service is talking to what service? Now it is, you know what service is running inside of where right so I can see all these different pods and containers running inside of these individual notes.

A

Sorry I'm gonna interrupt you. Can you make it.

A

Can you make it bigger.

B

Oh, oh sorry, yeah you were cutting out I'm.

A

Blind I can't see it there we go.

B

Does this help all right.

A

Yes, I'm going to say that's good enough, we'll let the audience chime in if they need it any bigger gotcha.

B

So yeah um yeah, so so what this is more of a physical layout and I can see some. You know important metrics uh things like you know: CPU and memory utilization at a high level uh for each individual node, uh the node names, uh some instant details, IP address status of the note and then I can see all the workloads running inside of it. So again that quick view that we saw earlier is still available. You know, basically, throughout the entire platform.

B

I, can click on this nginx pod and back inside that ninja next plot I'm. Now inside of that quick view more in the application context converted onto logs Etc, but a couple of cool things on here.

B

So um if I want to see the configuration of the specific node again no need to log into you, know your cloud provider of choice or your virtualization platform of choice, um no need to go into Cube CTL get nodes, Dash, oh yaml, or you know whatever, and take a look at your nodes or even log into a separate UI. You can just click on the config piece and now we're giving you the configuration for this particular node.

B

I'll the labels attached to this node, whether you're using those labels for like affinities or uh you know whatever they're all available right here you can see the IP addresses the runtime version of the of the container platform, uh the IP address for any for any network, overlays, uh cubelet versions, Etc Right Storage attached to it all. This is visible in context.

B

um You can see the same thing that the full-on yaml file- that's configuring, the node and powering that, and also one really cool thing that I, like is you know, I, should actually before that I'll call it metrics too, which you would expect right disk utilization. I o time no regularization, CPU Etc, all those metrics that you would expect to see there are available in context, but now the kind of a really cool view is our uh our balancing View.

B

So our node balancing view what it does is that it shows you at a uh CPU memory and disk level. What your consumption of of resources is right. So here's here's an example that is actually kind of interesting, so I'll I'll just look at CPU side, but you can look at memory.

B

You can look at disk um so, for example, here in this online boutique namespace, we have the load generator right and this little generator inside this container is main, um but uh you have a request set to uh 20 uh 20 uh Milli cores with a limit of 500 Milli cores right. So that's kind of interesting, because if I look at the value of utilization, it's pretty low right for the current, isn't nine Milli cores one of the average is nine million cores and the one day max is at 14 Milli cores.

B

You know this might be an opportunity to drop your your limit to maybe something lower, so you can make sure to schedule more workloads on this particular uh node right or, if I, you know, sort by a couple of others. Let's see, um let me find something: that's maybe a little bit even more dramatic, so here the payment server right now, I'm going to look at memory for the payment server.

B

So if I look at memory, I see that the request and limit are kind of pegged right at four gigs, but I see that you know it's pretty much static, right, 940 Megs on average. So you know we want to have our clusters, not you know packed to the brim, but you know moderate moderately to you, know: semi densely packed this. This is probably an optimization opportunity to make sure that we're squeezing the most that we can out of our clusters.

B

um So this is just kind of just kind of one view hey. You know we can probably drop this request down and limit down, maybe to a couple of gigs cut this utilization in half, and now we have some work for an additional workload on this.

A

B

A

A

And you might, you might get there so feel free to tell me to um to just wait patiently, um but I'm curious as to um when we're looking at these things. Do you have like a fleet viewer so, like let's say, I have 17 clusters that I'm looking at? Do you have some sort of fleet-wide perspective on some of these same things,.

B

A

A great question.

B

The answer is currently unfortunately: well, it's not. No. We have a B1 release that I actually don't have in here um of a plus of an environment-wide view where you can see multiple clusters anomalies across clusters.

B

um You know I actually should have, should have uh brought the brought that environment up um to show you, unfortunately I, don't have it up right now. uh But yes, we we've just very recently released that view. um It is something that we've been actively working on. That is kind of a big piece of our roadmap. A lot of our a lot of our customers just flat out have you know: lots of environments, right lots and lots and lots of clusters some of them.

B

You know some of them have two: some of them have two enormous. uh You know clusters, but then there there are. You know the opposite kind of end of that, where there are, you know, 50 100, 500 clusters, but they're that are small, so that view is, uh is in active development. There's a V1, um uh if you guys are interested in that you know, I can follow up with you guys and show you some screenshots Etc of an environment running that view, but not active in here. Yeah.

A

I mean that's, that's fine, it was and and yeah I mean I think we see obviously there's there's a Venn diagram of our customers and there's like, like you, said, there's there's there's people that you have in your customer base or or openshift users um and yeah. We see the exact same thing. Some of our users, have you know one like 55, node cluster and some of our users have like 10 five node clusters right like this and then anything to either end of that Spectrum in between Beyond.

A

um uh Who knows what? Who knows what people are doing? um It's every single person, every single company has something else that they're receiving in terms of value from the platform. So yeah it's one of those things and I think when you think about a fleet, and you think about like Fleet, Management and anomaly, detection right, I know one thing we hit in the observability and our our spaces. Well, some of these clusters are Dev clusters right.

A

These are development environments, um and so, when we're looking at them from a managed perspective, what might appear like ordinarily would be very strange. Behavior is perfect, like perfect behavior in that type of a context, so I know that's one of the most interesting paradigms. We've had to deal with from the reliability engineering perspective on that platform is okay.

A

Well, yeah we've got to keep some our stuff up and running and so forth, but also these are development clusters, so sometimes we're going to see some stuff, that's kind of weird um and I'm, not sure what I'm not sure. What from like your side perspective on the on the customer interactions on that type of thing looks like in terms of how your system interacts with those types of environments, but I know on our side.

A

um It's resulted in some very enlightening conversations with their customers, where we really get to learn about just very cool things that people are doing yeah.

B

And absolutely I mean it's an interesting, um it's an interesting scenario to run into right, so yeah you're right, especially especially with kubernetes and basically any in any, like modern platform, where you're doing a lot of a lot of get UPS. A lot of you know constant releases.

B

um You see interesting, Behavior or behavior that in production would be. You know, level one priority, call the entire company and get them on the line kind of scenario versus in the dev environment. Nobody even cares right and it's and it's just that kind of Paradigm. So you know one of the things we have a couple of interesting features um that that hook into that I mean it's.

B

It's uh uh we have machine learning, uh anomaly, detection right so when your behavior is out of whack right, it'll, all kind of skip into the the alerts here, um I'll find something that is ml based. So, for example, here we have some Network metrics that are not normal in one of our containers right. So, in this case it looks like we have a request, count uh Spike right, which is abnormal uh apparently for for this particular container, and not only we're not looking at just one metric right.

B

Our machine learning is looking at a bunch of different metrics uh parentity right in this case it happens to be a container so you're going to see more infrastructure. Metrics things like uh request, counts memory, utilization file, system, utilization, Network utilization. You know bytes in bytes out requests that request rates, error rates, all those kinds of things taken into account together, right and I'm kind of sidetracked and I'm going to come back to the question but yeah.

B

All these things are taken into account together, um and so when these, when these metrics behave anomalously, even if they're not picked right, where you don't have to wait until you're at 99 of memory or 95 of memory to trigger an alert if things are abnormal, if you're having 50 requests and you're typically, you know after three requests per second you're at maybe 20 Megs, but now you're you're at um you're, 50, 55, requests per second and you're at you know: five gigs of memory. You know you're 100 times over.

B

You know whatever that might be. Plus your file system is going crazy when it usually doesn't for that level of load. um You know we detect those metrics, how they interplay with with each other and then we'll we'll fire an anomaly.

B

um But my point of that was was to say that we have like, for those other environments, um for those lower environments where you have like erratic Behavior you're, maybe going to send a sudden surge of load Etc, and you don't want it to be detected on anomaly. We have what we call lightweight binary feedback um on on whether this is really a problem or not, and it's actually really valid for production too, but we see it.

B

You know you know used often in Dev, so I have a machine learning alert that we were just looking at and we can we're going to dig into some of the more um cool pieces of that. If we have time um but I can just say, hey, you know what I hit this minus sign. This is not an anomaly. This is normal expected behavior and that helps kind of tweak the sensitivity of the machine learning to those kinds of scenarios.

B

Inside of those lower environments, like you said, um to help avoid you know excessive noise for what is perfectly normal, behavior in that type of environment.

A

Yeah yeah I think so in one of the previous streams, um one of the things actually um ml, around, alerting and and Ops and so forth, and actually come up as a question, um and it was one of the things that we talked about there and I'm sure you. You can fact check me on my hot take here.

A

um I've been in order for ML to be truly effective. You need to actually first Baseline understand the behavior of your workloads in your environment before because it requires the training and the tuning so I'd love to get your hot. Take on my hot. Take no.

B

That's absolutely true: I mean I, mean and what's interesting. Is you one you're right on point and then you know going even further. um You can't just learn once and set it and forget it right. The reality is that you need to be constantly learning and learning and learning. You know your your ml needs to constantly learn uh and and constantly updating those models, because those models can quickly be out of date right so, for example, obstacles by default. We learn like every 24 hours, we'll retrain our ml.

B

That can be shortened like if your time cycle for for relearning needs to be at five hours or whatever you know that can be sort of by default every 24 hours so we're not, and and also we're, not moving that model over. Let's say you have a um you have some sort of uh you know you have an Apache application and we are, you know, learning on that and then we're propagating that same model to all your other Apache instances in in the cluster.

B

That's not what we do so, each application each deployment has its own model, that's learned! So it's we have a model template. We know what metrics to look at like I showed you, you know. Different you'll have a different set of metrics, of course, for like a redis uh instance, then you'll have for a container than you have for a node. Then you'll have, for you know a database, so uh those uh those models are are built to look at each instance of your micro service and learn from there and then constantly be learning.

B

So your hot take is actually a good hot. Take right, it's it's not necessarily uh something to be argued. It's absolutely on point.

A

um This reminds me there's something: I've said before I said it before I'll say it again: I'll probably say it until the day: I retire. The software development life cycle applies everywhere across the board. Apply it to everything right, apply it to your get-offs, apply it to your ml, apply it to your documentation, apply it to your operating procedures, wherever you are doing a thing that last the point of the software development life cycle. Of course, I.

A

Think I'm speaking to the choir here right is that that you, you take what happened after you did the design and implements and all that other stuff, and you you put that back into design again and you take that information and you redesign based off of what you've learned um so that last little part of the feedback loop right just everywhere across the board.

A

Every aspect I apply the software development life cycle to the way I wash my dishes right, like anywhere that you can do it observe, take your observations, put them back into your design phase and one of these days I will be better at spatial reasoning and be able to effectively load. My dishwasher, as well as um as well as the the person who helps me with my housekeeping, does uh today's not that day, she's still better at it than I am but like I get better every time. I I grew up with a dishwasher.

A

I should be better at this I'm just I'm. Just not she's like her Tetris game is just the best I've ever seen, uh which is a huge digression, but.

B

A

Just straight up like silly over here right now, I'll, let you get back to it! No.

B

But yeah dishwashers are a sensitive topic in my household I'm like why. Why does my wife wash the dishes by hand if the dishwasher is there and so don't get me started on that track? Oh.

A

My gosh I have uh there are so many reasons why that is. uh We could have it like you're like follow me on Twitter, for my thoughts on dishwashers, I will post them after.

B

All right, so, uh okay, so let's kind of um come back into here, so one thing before I. Finally, I could probably fully show off the anomalies, and you know what I'll show a couple of anomalies as as um as an interesting piece. So like our anomaly, detection is based on. As you saw earlier, we have machine learning anomalies, um we have rule-based anomalies just we.

B

You know we we play both in the app space and the infrastructure space, um so things like, let's say: you're you're focused more on right now on the kubernetes side right, you have a deployment. The ad service deployment is broken. It's there's no available replicas for more than 120 seconds. Maybe I'll switch back to live mode, just to uh kind of wake us up again. um So I have a deployment here and it's failing. I don't have enough.

B

Replicas I can click on it and and uh and now I'm seeing okay there's you know you, you have less less replicas than what you intend to you want one. You have zero a bunch of important information about. You know the namespace that it's part of uh The Entity that we're looking at, which is the ad service I, can click on it and go back into. um You know this. This config snapshot for the for the 80 service.

B

um Again, this is this is at the deployment level. I can look at the configs in the context of the anomaly right. So this is the config for that deployment itself. um I can look at metrics I have uh what we call time travel which but I'll focus on just the kind of the shortcut of this. If I click on Analyze This is now our root cause analysis, engine kind of taking place right.

B

We identified that there was an issue with a deployment, and now we bring up this is Chicago diagram or what we just call our fishbone RCA, and so this is very contextual to the type of anomaly that uh that is present right and I'll. I'll show you in a second what it looks like for. Maybe like a machine learning anomaly. That's metric based versus this configuration anomaly. You'll see different categories: pod scheduling, node failures, startup failures, runtime failures, so you see there's some issues with scaling up scaling down.

B

There's some back off restarts, but you know pretty interesting here is that we have an invalid image tag all right. So if I click on here I'm bringing up an invalid image tag I'm trying to get this ad service name I'm. Looking at this image tag, if I'm, even you know, if I want to be a little bit more adventurous, I can actually you know, go to this. Go to this uh uh repository and uh I'm.

B

Opening up that repository and I can see that I am trying to pull the tag called the v040 underscore error, and there is no tag in that repository for that image. Called v040 error right. So that is clearly why um you know this is bringing me at root cause as to why um I can't pull the image why my deployment is broken. I have a bad image tag, so you know we take. We take.

B

um We take different steps in our in our causal engine to really um walk through all these different pieces and understand all right. Is it you am I out of memory and I can't? So if, if you are, um if you don't have enough memory to schedule a uh a pod right, you might get one of these triggered insufficient. Cpm would say: hey you want to assign 500 Megs uh of memory, um but you're completely out of memory on all your notes. So uh you know you can't start up.

B

So those are the kind of things that we that we help uh identify the other. The other thing. Let me just go back into my alerts here, um there's other types so, as we saw the machine learning alert uh was triggered. So if I open up this machine learning again and now actually go to the analyze tab, you'll notice that the categories have changed right, so I have like memory and CPU and I can actually see why rml fired right. So um this demand side is incoming requests on the incoming side.

B

Everything is good, but on the outgoing request, right I have a spike a 20 111 Spike on my outgoing requests. I have some increase in file system utilization, 204, and so these are all things that our machine learning has detected as of Norm.

B

One of the cool things that we do is instead of having you build all these custom dashboards, which we do have the capability around, but we really try to shy away from that because, as mentioned earlier context, is king So, we try to bring the context to you and feed you what's going on so, instead of going to a different dashboard or you know, building a drill down all right when I'm in here, you know drove me down into this dashboard, we're automatically dashboarding these important things for you right again, it's not losing the context.

B

I can see my response time um during that time frame of the anomaly I can see. My file system reads how they're going up and down um I can see the CPU utilization. I can see all this data, basically in the context of the anomaly, and if I want to go even further in this detect tab, I have a bunch of data that I can access if I want to see logs. For example, I can click on logs, but it's not logging, because it's not even starting up.

B

So that's you know, but when you're having, when you have an issue um uh and and uh in your application, you can run into the logs, you can drill down into the configuration, as I showed, also drill deeper into metrics right. If I click on metrics now I'm going to look at the metrics, and you can see during these time frames or the anomaly, these you'll see these red bars.

B

um You know what was happening on this particular instance. Now there's some there's some other pretty interesting um things that I can show, but I do want to show you something that we recently came out with um and before we run out of time and um and one of those things is uh the tracing capability that you know, that was your very first question of you asked about APM right, so we recently introduced APM into our platform and we do something.

B

That's really really cool um and um before showing this actually I I have a I have a slide to really explain the kind of differences between what you saw earlier in that automatic network discovery map and and what the tracing piece is, so that app map, which is really really cool, all right um uh is has its shortcomings, mainly that you can't look at the code level right. That's just for code level. You require co-level instrumentation, which is a distributed tracing component that the open, Telemetry libraries. You know Jager Tempo Etc.

B

So this is what you saw earlier that map it's service, a toxic service, B toxic service, C, toxic service D. If you're you know talking in HTTP, we can actually see the URLs that are being called um between each services and performance, and we have a screen for that. But then the other kind of uh uh side of the spectrum is uh traces right. So um once you enable traces on your application, you're going to see all these individual requests again, you're you're.

B

Basically, your business flows uh traversing to the system, but here's what happens what's happening so for most tracing platforms and again this this was my bread and butter for many years. So I I can tell you that I dealt with this day in Day Out. A trace gets named basically on its entry point.

B

So if you have maybe like a search right and um so so, let's say you have a search transaction flow right, even if it takes branching paths which you would see here that it's going to a different place, since they all started that search um they're, all just going to be aggregated as a search.

B

So when you're, looking inside of your of your system, you're just going to see a single, you know, transaction flow called search, and the problem is that you're actually taking different paths, which means this could likely technically be categorized as a different platform. Here, you're hitting you know, operation V1 versus here B2 that could completely change the response time um you might be hitting you know, basically a different microservice for a different um uh or a different function. It doesn't matter, but because of the way tracing is designed, they'll all get aggregated into one.

B

So there are ways where there are the platforms where you can, um you know typically I would have to when I was doing delivery sit with the developer. uh You know find their. um You know, talk to them and say: well what can we? What can we split out these these transactions on? Because um because my search, this one is mobile, but this one is desktop so how to identify between mobile and desktop searches. Well, we have a request parameter.

B

Why don't you put on the request parameter or we have some method argument that gets called? Why don't you just put on that? So it's it's involved, it's possible, but it's involved. So the cool thing that we've done is without having to involve developers without having to do um you know much of anything once you've got traces being fed into the platform, we will automatically identify the different service operations service, operation, pairs that are being traversed so even though they both the search, starts here at the same place, we'll identify that hey.

B

This is hitting a different operation. This is a different, technically a different transaction, so what we will do is we will automatically split them up for you and instead of seeing aggregated metrics for just a single transaction, you will see the metrics for the transaction type 1 the what we call a trace path and then Trace path, two all right and what that looks like in practice. Is this um I'm going to go into our trespass view here and so I actually renamed these so I'm?

B

Actually um uh I'll show you that I'll just reset the name here and reset the name here. So you can see right if I see this receive cart and receive card, we actually have. It looks like three of them, um but if you notice on the trace path, what we've done is we've identified that this is hitting front end, seven times and one time to the product catalog service.

B

While this is hitting the receive card three times and only one time to the product catalog service, so I can click on it and see a view right here. My seven calls one two three four five, six seven and then, um and then this one call versus this other one right if I go back to all Trace paths and then take the reset any case, um this other one with only three right and I see that it's only making these three calls. So those are separate transactions.

B

Most other platforms would aggregate them and you'll just have a single receive cart, um but what I've done is I broke them up. I'm, not sure why the reseted it I didn't say, but um but I've broken them up. So I know that one of these is IOS and one of these is desktop, even though they both start at the same place. You can see that there's different performance right. This one is 3.94 milliseconds. Well, this one is at over 28 milliseconds, so performance profiles are different.

B

um The request counts are different, um but now now you're not getting aggregated data, because if, if let's say my cut off for a good performance, is 25 milliseconds?

B

Well, if you merge these together right, assuming that it's one call in one call, if you merge these together and then take the average you're at 28, plus one you're at about 31 milliseconds, average them out and you're at 15, milliseconds and you'll say: oh yeah, all my transactions are well within range, but if you actually break them out, you'll see that yeah, your desktop searches are fine, but these other searches down here are above what you're accepting SLO is.

B

So what we're doing is that we're helping avoid false negatives on on uh on different traces and I know I'm practicing that a time um there's a bunch of other stuff that we could show. We have automatic anomaly detection on different Trace paths. You can go back in time and look at your configuration and we have a bunch of different root cause analysis pieces but um I'll stop there.

A

Awesome yeah, uh that's I, actually so I was recently having a conversation. Why is Hillary so passionate about this right? I was recently actually having a conversation and the the person said to me. Like you know, from a nursery perspective, um you know what was what are just like some chores that we should be doing on our on our cluster and I was like. Oh man like that, I there's a whole basically I told them.

A

I said: there's an entire conversation that you and I need to have about observability, because there's not you know routine maintenance and so forth. Right! That's! That's your sis admins right, more or less most of the time natural system is they're. Gonna handle that but like what are the, what are the things that an SRE does right?

A

We we should be acting on this data that we see here and applying that back into hardening the systems tuning the systems you know actioning on this data in some way that makes Services you know, platforms, workloads, applications whatever it might be, that that you're, interacting with, on whatever degree, better right absolutely, um and so that's why I I the timing on this is like we'd already had this scheduled to have that conversation. I was like this is such an important topic.

A

We a lot of times, especially especially in a profession like SRE, where we do spend quite a bit of our most exciting work hours, trying to figure out what's going wrong in production like what's, what's actually happening, what went wrong what fell over um and that really is just like the the smallest, not smallest. This is just a small part of the bigger holistic picture, which is, if you can get the current state of your application or your service based off of the metrics and the data that you're pulling from it.

A

Then there are things that you can do to make it better, um so I yeah, we are coming down to end of time. I greatly appreciate you uh coming on here, showing this off uh talking to people about it.

A

um We've put some links in the chat um actually for um for uh for checking out more about Ops, Crews um I think I saw um I, know. I saw that you all have a free forever tier, which was pretty darn cool and it's displayed on the bottom of our screen right now, actually um and yeah. So there uh is a um you all have a presence on the kubernetes slack as well uh We've linked to some of that in the chat uh and yeah.

A

So I hope you know viewers here um go, you know, hang out interact, ask questions, uh tell them you heard about them from me. So I get brownies points, um there's no Kickbacks just brownie points, um and uh maybe maybe uh maybe we'll have you all back uh again someday uh to talk about additional additional stuffs and things yeah.

B

That would be awesome. Thank you very much for having me on Hillary. It was a lot of fun.

A

All right so with um that said uh just a reminder that next time we are back, we will have Christian Hernandez again, uh which I am super, super excited about and uh we'll be discussing the Argo rollouts and with that Johnny I'll, ask you to uh to roll us out and remember folks: choose your technical debt wisely. We'll see you in two weeks.

A