Red Hat OpenShift Developer Experience Office Hours, 14 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Developer Experience Office Hours: Datadog

Description

OpenShift already provides you with some cluster metrics, but not much at an application level. In this session, Joel Lord from OpenShift will be joined by Ryan MacLean to see how to install Datadog in an OpenShift cluster and see how it can be used to find issues within your applications.

A

Good morning, good afternoon, good evening, hello, everyone welcome to another developer experience office hours, I'm very, very happy to be joined by Joe, Lord and Ryan Jarvan, and today we also have a special guest Ryan McLean, the one and only Ryan McLean from data dawg. He is here today to talk to us about all things day to dog and open shift. It's what I'm hearing where aids? Let's, let's just do a quick round of introductions, so everybody here knows each other and Joel. If you want to start off, that'd be great yeah.

B

I'm gonna get started, hi everyone, I'm, Joel, Lord, I work as a developer advocate on the OpenShift platform here at Red, Hat I come from a developer background. I started using some of those technologies like OpenShift, kubernetes and even containers not very long ago, so I'm still kind of new to all this world, then I come from a very developer background really thing that I was trying to do recently was to get data dog and to see how it can help me to kind of see.

B

What's going on inside my applications and I was trying to set that up on OpenShift and I had a few issues. So that's when I actually figured out. Why not reach to the experts and I reached out to Ryan and I said hey? Why? Why don't we actually do a twitch stream, so I can actually learn as the same time as everybody else so I'm glad to have you Ryan? Why don't you introduce yourself yeah.

C

No problem so I'm Ryan McLean that data dog, my official title is a tech evangelist. We were laughing about that. A little bit earlier might change in the near future. You never know, but I've been been in the industry for about 20 years, I started as like at bench techs all the way down from you know, working on laptops and stuff, like that, all the way up to helping people out with your infrastructure, both large and small these days.

C

What I try to do is help people with getting things set up in the cloud and starting to monitor their apps as well as getting that that big picture kind of thing clang, so so, I think when you and I talked we talked a little bit about so the difference between like infrastructure, monitoring and application monitoring and how the two might work together and that's that's kind of where a lot of people are these days as well. So that's this one trend, oh yeah,.

B

Exactly so, I was my main question was why why should I actually use data dawg when, like I I, do have all of those metrics, and it tells me stuff about this usage and and memory usage, which you know, I, don't necessarily really care a lot about as a developer, but I was like well. Why would I even bring data doc to get more metrics right, yeah.

C

And I'm, not in so you're, not gonna, get a hard sell here, but the main thing is like for infrastructure monitoring: it's it's not a commodity, but like a lot of people, do it right what you'll see, at least in terms of coming from your background, like a developer, a background, a lot of what we can do these days is is almost like monkey patching code, so so like injecting ways of tracing or maybe getting telemetry out of your code as it's running in the cloud or as it's running in kubernetes or even on the VM or or hardware.

C

So you can see things like stuff like function, calls or method calls, or even calling third-party api's and get metrics on those, but not just for like those calls in aggregate or maybe on average, but also how they might depend on each other. So if you've got micro services like if one is calling another, what that looks like or if you're, seeing like a lower rate of calls for your team service, but it looks like everything's flying on your end, sort of being able to see what's upstream, that can be really valuable.

C

I think that's where, like that, the APM crowd kind of comes in right and that's that's sort of our place. So the infrastructure side I mean to be honest, like you'll, get a lot out of a default Cooper days, and so you'll get a lot out of a default cloud. Install like a VMware, just install a red heavy em install all those will give you like they, your CPU, your memory or disk IO stuff, depending on being able to use that as context for your applications performance. So what the business really cares about right?

C

Can people actually check out, as opposed to like it's a server app I think is maybe more important, so sort of flipping that problem on its head and we get it from like the the business angle and then drilling down into your CPU memory and metrics. Like that, so it's just a different way of looking at the same stuff. It give me a bit more context. At the same time, yeah.

B

Absolutely when you showed me the other day, this APM stuff, I was really impressed and I was very excited and then I tried to install it and and unfortunately, I wasn't able to get it to work. To be honest, I didn't try very hard. It seemed like something that was very straightforward, but I just had five minutes to test it out and and I had never actually got it got it to work. But knowing that you were gonna, be here, I figured you know. Well, we can just do it together.

B

That's that's gonna be even more fun and, and hopefully it will actually work yeah exactly.

C

Yeah, yes, so now I'm here, let's, let's just talk about like if it doesn't work so so one is on I. Think one of the things you're trying to do is install something that's pretty new to us as well. So there's there's a few methods of installation inc in kubernetes if you're not used to it. Basically there there was a whole push behind a lot of people, got home, working and like to sort of use as a package manager for kubernetes we're moving away from that to operators and I.

C

Think redhead is as well, but on the operator side, I got to admit where we're still pretty new to this area in terms of having it being customer facing, even though we use it internally and because of that, its beta, so I think some of the problems you were having is because its beta, we can try it out again today and figure out. What's going on there.

C

The long and the short of it is the the install into openshift is very similar to a vanilla, grenades install, but there's there are some things like roles and permissions and privileges I think are specific to open shift to allow us to get some of that. Like networking, for example, I think we need a little bit more and I have a feeling that when I looked at the operator, I didn't see a lot of that in there. It could be that just needs to be massaged a little bit.

C

I also know, and I was hoping that there would be a newer release of the operator coming out. It hasn't launched yet, but it should be out in a little bit we're trying to do an internal conference here. So I know, we've pulled a lot of resources in that area, but the operator definitely is our focus and India is where we want to be there's anything wrong with helm or default manifest it's just what we're using. It seems to be a big push in the industry as well, so we're gonna follow suit. Yeah.

B

Cool all right, so why don't we do that today? We can. Maybe you can give us a quick, quick tour of data dock, just to get a look.

D

B

It looks like and then I do have a cluster I think I've shared the credentials with you, so you have an application running and, and we can then try to install the tracing as part of that application and see where that leads us. No.

C

Problem so I'm just gonna quickly, log into like a demo account here. If that's all right, riff and then I can I could share my screen. So we've basically go ahead and I'm sure this happens with red as well. Look at them away.

C

Infrastructure is set up in the cloud we can just sort of log into and show people you know just like a normal application and and what that would look like so something with quite a few micro services and quite a bit of interaction between these that it allows us to just talk about how you can drill down. Look at look at different micro services talking to each other kind of thing, I'm just going into it here and I'll see if I can drag this window over. Let me know if the screen res is too high.

C

I'm on a 4k display and I know. Twitch doesn't always like that, but.

C

Yeah I don't know about you folks, but I find that 720p, since we, the right so I'll, see if I can do that. It.

A

Yeah, there's just like happy medium between able to do work and able to see work on stream. Yeah.

C

It's gonna hide the there how's that.

E

Font, does it look.

C

A little bit small yeah well.

C

All right once I can hear all the pop it at this yeah we'll get some 720 going yeah. You know people will have different resolutions on their end as well, plus int, which I noticed if you've got it in like a little window here with it the faces on the side, it can be pretty.

D

Scaling, let's plate.

C

This big flare chair and see: okay,.

D

C

Can sort of read on my end? Can you please read that there that's better, better good? This is some inside baseball here it on the tree, sensei, okay, so I just logged in you can see my email there and you can see that I've got some stuff running I'm, just gonna quickly, minimize this sidebar because we're not getting any of it. Basically, when you log in you can see like all your services and we're talking about a p.m.

C

today, but we also mentioned infra I think that the main thing that people want these days is to see that application performance or that service performance or the relationship between the two and less so the infra.

C

So we'll just talk about like some of these services here and specifically what we were talking about, which ole is just tracing I, guess in in particular so ways of looking at, and this is like a like a 15 minute, but we can look at live trace here, so what's actually going on in in this okay, so it's life actually looking at. What's going on in this store and I'll just give some quick background here. So this story is imagine you've decided to start selling t-shirts with dogs on them.

C

Maybe some hats with dogs on them, some sort of dog branded merchandise. For a fictional story. It's got an ABS ad service. It's got a web store. It's got some. You can see Redis going through here. It's good, quite a few components. It's got some 80 authentication as well. If you want to just have have that stuff set up, so it allows you to look at like a bunch of services working together. You can see that there's, there's actually traffic going to the site right.

C

It's flying by there's errors that kind of stuff I'm just going to quickly pause this. So we can sort of drill down and a couple of these errors. So you can see like- and this is typical you'll you'll get like your your 200 or your 2x. X is 3 X X, maybe not so much your 4 X, X or 5x X, though they'll come through and generally what you'll want to do is not necessarily look at them specifically, but maybe in the aggregate or on the whole, so kind of looking at the whole thing.

C

So traditionally, what you do is you kind of go in and then just just gonna left hand side here, I got rid of those okay, so all those 200 certs are gone and now I'm, looking at the four hundreds and the five hundred here so specifically the five hundreds. What do we got and sort of I was telling you all is that we can actually click into these and then we can get it almost like a flame graph. So if you've ever used something like dtrace I in the command line to look at your app.

C

This is what they look at. If you haven't that's fine as well, but basically in this case you can see at the top here. We've come in and we've hit the the laravel service for pricing and in this case it's using a clustering, goop cloud, but basically imagine it's it's any infrastructure. You can see that it's look like like a third of a second to load. It up the kind of things which it takes well, I'm just clicking.

C

Like a nice juicy one here, okay, so basically what happens with the flame graph? Is you see the top coin come in and then under that you'll see anything else that it called, and some of these can be can be pretty involved. Some of them can be a little bit less interesting and basically you can see them on the left hand side here when there's multiple services, so just going through to see if I can get one. That's got like quite a few calls in it, so this one might be a little bit better again.

C

It hits level it goes into level calling itself basically for an action when I click on these I can actually go right into what's happening here and I can see these areas. So there's a guzzle, a cgb sort of exception and saying that there's a bad gateway, a kind of thing, but I can actually go all the way down into each one. So this is expressed, for example, I can get down all the way into the Express middleware and then all the way to the HP out get in this case.

C

It failed at the top, so a lot of the failures aren't gonna percolate down into the middle. We can go all the way down here and see that you know Express is trying to call another service and it's actually getting a find missing. Now you can see that it's then calling the middleware that actually goes down to Redis and says like hey.

C

This is not how you use a get call to call rid us kind of thing so along the shirt here is a you can actually drill down into each call for each service and within them. If it's, if it's a problem in the actual code, you can actually look at the the functions or the methods themselves. Now. Some of this is done automatically and that's great.

C

What I would recommend, though, is once you start using a service like this you'll probably want to start making your own metrics or your own business, metrics or things that are of more value to you. Then then, possibly like the box stuff. That being said, the others box stuff will get you started and get you started quickly, which is what we're hoping for today is to do what we call monkey patching, but a way to sort of almost side load your stuff in there.

C

So I think we were talking about node, there's ways of just essentially importing at the start of your file. You might need to you, know, rebuild your container and relaunch it or or to launch to it like a test cluster or staging cluster prior to just you know, Yolo eating it to product kind of thing, making sure that you got it working first. I put.

C

That being said, it's it's generally pretty straightforward in terms of kubernetes I'm, pretty familiar with that part and Joel I might lean on you a little bit on that on the note slide in terms of communities, it's the same of installing any other like any other helm charge or any other manifest applying it to the clusters very similar methodology. So that's basically it that's that's kind of like the big tour you can go from here into anything else. You can click on this I'm Pro.

C

If you want to, you, can actually go right into the hosts that serve this in this case is probably a container excellences gke. You can get it all. Those metrics that we're talking about before the infrastructure, metrics will come off. No problem and I can go in and look at the network traffic dis latency you can from here. You can start going into the cluster and stuff like that as well, and as you go through here, you can. If you need to correlate anything in here, you can do that as well. That's basically it so!

C

You can pivot back and forth between the the front end different microservices, different API is different, API backends and then the Peter clusters as well so allows you to hit quite a few different angles. If you want to drill all the way down into logs as well, you can do it that way. So, if you want down.

B

To it different I like how you can easily connect two to one to another and then to just jump in from from one section to the other. That's great yeah,.

C

B

I think that's where the industry is going.

C

To be honest, you'll hear a lot.

B

C

Abilities will sort of taking all of these and bringing them together. No people will say it means different things for different vendors right, you're logging vendor. It will tell you it's it's about logs your event, vendor away it's about events and for us it's more about grabbing the signals. So what signals can we give you that allows you to give you a big picture? I.

F

Noticed on the the previous page, you were looking at there. There was a section that was highlighted. Called tags, I mean.

D

F

Want to tell a little bit more I haven't I'm not familiar with. What's the significance of that name, I know there was also errors was a tab there, but you want to go into that. A tiny bit ya.

C

Know prom, so this one here you yeah.

F

Yeah, what is what is tags mean specifically yeah.

C

So a tag is it's made of data which doesn't really help it alright, but basically.

F

It's essentially on the request: yeah.

C

It can be so so yes I know so so. Basically, a lot of this you can see it's X request. Id is is actually how we tag everything together. So basically, as things are coming in to your services, we're putting things matter to say, as it goes downstream, how do we know what's what kind of thing the tags themselves like if we go into infrastructure here?

C

Some of these tags, like in in openshift, the tag might be like a project, I think we'd, probably a common tag, maybe the node that you're running or possibly, if it's, if it's somatic, a specific name space or a user's name space. That can be a tag as well in this case, I think most of the tags that you're seeing here are things like the environment. So an environment can mean different things to different places, but a lot of times it would be like your test.

C

Staging your product might be a tag so that it takes a key value just to just to clear that up so generally, a tag will be like what type of tag and what is the value of the day. So, in this case, it's a little bit small here. Also you can, if I can move in, but basically this clustered location here are the clustered name, for example our tags with a cluster ID. We can use those tags so then pivot into the cluster or pivot into the environment. That's a good question, like the tags.

C

Do allow you to give a lot of rich info in this case, since this data Doug, we we put a lot of tags on here right.

D

C

It's it's up to you to pick and choose which ones are valuable for your bids in OpenShift.

F

C

Of these are default, though sorry go ahead. Oh.

F

Oh yeah, all right, yeah I was wondering how much of this is automatically provided by data dog versus something that you might need to build into your model. As as your application monitoring matures right, yeah.

C

So a lot of this is done default in the cloud provider or I. Guess you could say default in the infrastructure provider, so in this case we're in gke. So there's a lot of gke tags in here, but if we're in AWS or in Azure or what have you we've grabbed those tags from that environment as well, but the the corollary here is to get that data. We need to actually have access to read from that environment. So a lot of times you'll have data dog almost almost scraping.

C

If you will the API from the cloud provider in an integration, that's just data dog to cloud provider without any infrastructure at all. So it's almost like a like a reader essentially did tell you, but.

F

That sounds like a huge advancement to automatically get this level of markup appended on requests as they fire through a distributed system and then allow you to, like you said, do that pivot and and then drill down with a lot more certainty to as to what you're looking at and what the related edge cases may be. If they're highlighted via these, these tags right, yeah.

C

I think here's the nail on the head there, that the secret sauce is sort of the pivoting right being able to drill down or drill back up tags are basically what allow us to do that that that being able to, if you read the papers, that they call it baggage, but basically, as your traces come in that the baggage that you add to the trays are those tags or those those things like the where the request came from or how long it took last time that kind of stuff.

C

Those tags really allow you to add that baggage to the trace and then go from there to get more info.

F

Awesome any other questions on this topic. I'm super into this topic, but I also had a other.

D

F

Up what happened to the hexagon based visualization? Oh it's.

C

E

C

Yeah for sure, so the selling.

E

C

That's your favorite shape, so, to be honest, I've been using data dog for almost 10 years now it feels like that's sort of impossible but yeah, basically, as it came out, I kind of get interested into it. So this is what I was used to when I when I started using it as well as a hex to you right. So so, where's where's, all my in fright and you can tell we've- got a fairly large demo environment here, but basically the the hexes are kind of a way of it.

C

The reason we use hex is just just so you all know it's just because you can fit more on a graph and you use hexes as opposed to like squares or circles. You can just get more on here. So that's that's why we use them, but it the hexes, do it. They basically allow you to look at your infrastructure as a whole on one screen pretty easily and then sort of just pick out the orange ones right. Oh the red ones or whatever your visualization is to see what's having problems but they're still there.

C

So there's a lot of tabs here on the left like when I started, using gaeta dog this left hand, side may have had a couple icons and and we're up to a whole bunch of them now with like sub subheadings here. So the hex stuff generally exists under this infrastructure. So it's the host map is what we call it so we're all the host is, but we've now got like containers processes, servers, server, less network network map, all that stuff, there's quite a bit going on. So one of the more common ones.

C

These days that we like to talk about is sort of like the network map. So how do things relate to each other now? This is a bit of spaghetti here that I'm showing, but if we're going to look at those services like how do they relate to each other, this is. This is maybe like the new way that the new hex map, if you will like the the new APM kind of way and and again like there's a there's, a few vendors out there.

C

That will do this as well like I'm, not I can't do the hard sell here either but ways of relating your services together graphically- and you know the red green right. So if something's red, let's stroll on into it, let's see what's going on anything but I'm glad you asked cuz, yeah I might have just glossed over it and said: let's talk about APM today, but this does allow you to certainly drill into your in for us. Well, if you want to look at it from that aspect, I.

C

Think you might commute it there I just looking at that twitchchat, though, to see if there's any questions now.

A

There's a lot of back-and-forth about your bitter things, but you know no specific questions regarding a topic at hand. I mean folks tracing is huge. Right, like is the the capabilities. I think can be a little daunting, potentially maybe for some people right like there's a lot of stuff that you might have to filter through to find what you actually need, given that like moment context whatever it is you're looking for. So how would you help you know kind of people just getting started with Tracy and try to figure out like what is I mean?

A

We know golden signals. We know sre practices. We know all these.

D

A

With data dog right like where are they gonna jump in first kind of going on there Tracy journey yeah.

C

So I think the main thing we've got is like the the ability to quickly get you some visibility essentially quickly get you started with your cloud provider quickly get you started with your code.

C

The ability to monkey patch kind of helps out there I think the main thing here, though, when it comes to tracing I, there's a difference between what you can do automatically, which is certainly good and allows you to like drill into your code, look at speeds, but really it's almost like you want to take this problem and turn it into like a business problem or a product manager problem or like stakeholder issue.

C

So it's almost to say like if, if what matters to you is daily active users or monthly, active users or or signups or check outs, what in your code, do you need the instrument to give you that visibility and then how do you trace from there? So how do you? How do you put like a tracer bullet into that specific area that code? So if it's daily active users may be signups is something you want, or maybe it's like front page hits or.

D

C

Like that, like that yeah, that's so metrics, that's important to you! Sorry go ahead like.

A

The the most important part is, you have to know what metric is important to you. Yeah.

C

And that could be hard, but a lot of times those conversations involve sitting down with your stakeholder and saying you know if we had to put a KPI on our site. What would that keep yet like if we had to have one KPI? What would that look like a lot of times? They'll give you you know four or five and they can say okay if we were to wait these, which should we wait higher or if we were to score these like it.

C

If you know a plus B, plus C, plus B plus C equals X once we need to wait them for to get it and I'm kind of I'm being generic here. But if we were to talk about like this site, for example, the store site, one of them would be like how many checkouts do we have in the day like how many sales right, that's pretty common for a sales org, how many and if you're, gonna look at it as a funnel right like how do people get into sales?

C

How many people did we get into the site to give the sales and then between there?

C

I might be like how many people added stuff to the cart, and then it might be how many people actually went through and added their payment info kind of things like what does that happy path look like and where did they go through, and sometimes it can be talking to like your UX team, about what that path looks like or what story it is, or what branch they're trying to send people down and then once you've got that kind of done.

C

I'm kind of often left field here, a little bit, but sometimes when you've got that done, it allows you to do a little bit of almost like testing like like. Let's try this new story and see if it works better or let's try this new landing page and see if it works better, so allows you to experiment as well when you get there, but basically the long. The short of it is to figure out what's important to your business, a lot of times you have to talk to the business. Basically right.

A

C

Didn't race from.

A

There yeah as a as someone that came from like a DevOps background right, like I, cannot emphasize enough the need for developers, ops, people and the business to talk more right, like you're.

C

A

You're all after the same thing, but if you're not all talking about the same thing and you're, not looking for the same things and you're, not finding the right signals to indicate success or failure in some instances, you're gonna be kind of lost in the sea of you know. What is it we're trying to do here so having that conversation and getting that balanced as far as what you're looking for specifically is vitally important.

A

So even with this stream right like we have certain measurements that we're trying to hit and certain bravery to do so right, like there's, there's goals and then there's indicators of success, and you have to kind of lay those out for every project you take on yeah.

C

You got it and you mentioned yes. Are we there abyss well I'll touch on that, but, like then, there's also like the downtime, pretty the expected downtime, where the times of things fail. Maybe your planned maintenance, but there's some of that too, and and when it comes to the business like it's the same thing, you gotta communicate like what your your error budget is or or like what the reputational damage of five minutes. Downtime during Black Friday might be versus. Yes,.

D

C

In in March some random day kind of thing right, it's it's totally different, but at all it all kinds, it's sort of packed together. You say so I'm, honestly glad coming from like a few different angles in the business that we're starting to realize that you have to have shared goals and you have to be able to understand the indicators you're using or share in the whole company and you're all working towards that same thing.

C

So, if it's more sales, then everything should be enabling that if it's more viewers on the twitch stream, then everything we do should enable that as well I'm sort of like working towards that one or those two were those three KPI is to make sure you're there and I know. Kpi is kind of a word that people don't like, but basically those indicators right. Those things.

A

C

Tell you what you're doing the way to measure yeah.

A

Kpi see what you want about, you know KPIs, and it's it's more so, just like anything right, DevOps sres, the way you implement it right. So if you have a negative experience with KPIs, it's it's sad, but it might have been the way that this was done. So the the idea behind having those indicators are critical right, like monitoring, is like that base level of this Ori pyramid.

A

You know having tracing as part of that scene where certain people are falling off. The cart potentially are falling. You know falling out of the buying process or falling off the website or whatever it is that metric. It is you're looking for right, like they fall off the stream. They turned out by mail, boring me whatever, but yeah the the the those those metrics for success are critical and and having them. You know indicated and traced and observed, or vitally important, yeah.

C

And I see Ryan's, adding some some stuff here on the acronyms. That's great cuz I often do forget to define like what Nestle is. What missile is is I tend to its acronym super bright, but he he's got a good point here. So it's it's important to say, like a solo, is service level objectives, what you're trying to achieve and they're informed by the SLI. So when we're talking about those indicators as opposed to the KPI, is the SL is of the service level indicators thanks, Ryan I totally apologize for not deploying them.

F

Someone was already kind of, we were chatting about it, so yeah.

A

You know: that's the best part about these office hours is that we can all kind of tag-team. This.

C

Okay, I'm honestly happy to see the way our industry is embracing business and talking about share goals. I we talked a little bit about da use and ma use, but like what a daily active user actually means, is it visitors to your site? Is it signed in users? Is it unique visitors to your site right.

D

C

Know I worked at a mobile phone company today. I was here not today prior to today, so it'd be like is, is a daily active user on your phone, using like a tick-tock or an Instagram, or are they interacting with your mail app like what does that daily I couldn't I mean it's? It's really different, depending on the company that you're at so it's it's worth. Having that conversation, it's.

A

Always worth having that conversation, because each each bit each IC business organization nonprofit whatever it may be right like they have goals and if you can align somehow from top to bottom. What's the other part okay, ours are. The other part of KPI is right, like your objective objectives,.

D

And key results like.

A

They should you know, keep you I should align to that across the organization. I mean there's some tools that can help you with this, but there's a lot better outputs from people that work as an organization that then people that work in silos we've seen this over and over we've seen entire industries get upheaval to buy. You know DevOps and people just working better with data right yeah, so you know I mean going back to you know.

A

If you look at election data or any amount of data you can find and clean, so much information about it right like covert. Nineteen is a perfect example.

A

We have all these open source projects spinning up and they could all use tracing like this mm-hmm to see where their applications are, maybe not performing it as well as they should or to actually see somebody fall off because of that performance problem right, like somebody's coming in here for kovat 19, you know testing data right, like they're, trying to figure out should I get tested, yes or no, and if all of a sudden that you know checklist is abandoned, that's bad right, like that's!

A

A that's a public health concern so having tracing in here and having you know that, like hey, they hit this checkbox and then they hit submit, and the next page comes up it's taking too long because of this weird Java library or whatever. Right for this one page like you, won't know that unless you're actually observing and monitoring these things and doing it in something like data dog makes it a little bit easier and.

C

I think you you sort of you're hitting in a home here, but that's the other thing is that, at least in my career, a lot of what I looked at was like the mean or the average, and if it's just one of those surveys that somebody's not doing for their Tuco mid-nineteen test, just like that, you'll ignore it. But these days a lot of what we're doing is looking at like the 99th percentile. The ninety-nine point, nine, the ninety-nine point, nine nine. So let's look at those bad experiences.

C

Instead of just looking at like a like an average, because an average will hide everything right, it'll hide the bad stuff. Everything will look, fine, you might have somebody who's got a really bad time. It could be a VP testing. Your say right, that's happened to me before it might have happen.

D

Oh yeah, it was.

C

Like everybody else is fine, but the VP you got in the elevator and tried to load the site and it didn't load, and then they come back and tell you. Ok, your site didn't load and then you're looking through your logs and everything seemed to load, and there isn't is like you didn't you didn't get that hit like you, weren't instrumenting, the app on the phone, your instrumenting on the back end and the hit didn't come to you. So you didn't see it.

C

So some of this is like some of its end to end some of its not looking at the average looking at your actual nines. So your percentile sorry back actually being able to look at your 1% percentile, your slowest traffic, basically or your your most error-prone traffic and concentrating on fixing that, as opposed to concentrating on making your average lower, were hired right.

A

Right right, yeah and like tracing like this, can also tell you right where you might have Geographic issues too right, like let's say you have a pocket of customers in a region where you don't necessarily have a great you know fit. You know edge presence whatever you want to call it. Yeah like this can indicate that to you to an extent right like if you see a lot of traffic coming in through, you know like Tokyo, and you don't have a REIT. You know you don't have a point of presence in Japan anywhere.

A

That is like instant red herring right, like not red herring, but like something that you need to definitely the opposite of red herring right like that's something that I can address pretty easily in most cases right like spin up an instance or you know a region in Japan and off you go right, like you'll, you'll get better user experience from that. You know one or five percent with you know the the effort it takes to have what you have now right so and.

C

It's it's funny. You bring that because, like I literally had the issue so I've launched a couple services in China before and China has got a host of concerns that are a little bit different when it comes to cloud providers. One of the main is geography right, so so getting data into a region. That's not in the US when you're a u.s. centric company is difficult, so Japan's a great example.

C

I can talk about one from my past, so we in China that the main thing that we had to do is actually just set up a CDN right, like it's inside edge compute, but a way to get something set up locally, and that's that's how we found it was through tracing. So we could. You can see that all these users are having a good time and these 10 users, or these 10 million users are not having a good time what's going on with these 10 million.

C

This is a significant portion of the traffic and it's significantly slower. So, what's going on and that's what you found it's just around trip time from one geography to another can honestly be a killer for these. They.

A

Will kill you like a what was it a WS? They are not able, yes, Amazon, the actual like shipped the store. They figured out that like a handful of milliseconds, is that there's like yeah hundreds of thousands of dollars, somehow right, like yeah.

D

A

They've dropped those numbers, so they know that like if they shave off a millisecond here or there, yeah they're gonna see gains right.

C

And I can't remember what the number was, but was.

A

It was someone's really low. It was to be loaded in one second, less than one second page load times and like transaction times with less than 500 milliseconds, or something like that. It was something ridiculously low right and it's like if you're getting to that level, you have instrumented Freight for yes,.

A

Wonderful observation right, like observability and just the capabilities to do that is amazing right but, like you have to have that base layer. First right, like you, have to know, what's going on in your environment and I.

C

Think in that case, just to talk about it sometimes you'll say like hey that the page loads. Slowly and that's you know it's it's not good or bad. It's just what it is right. Yeah.

E

C

The nice thing about tracing or observability, or even just having these indicators in general is, is knowing that you can you've almost got a safety net, so you know what the baseline is. You can then say like hey: let's start experimenting: let's start lazy loading I start putting our fonts in a CDN start, making our images a little bit smaller than they are.

C

They don't need to be 4 megabytes, that kind of stuff where you can start running those experiments and like blue green deployments and kubernetes right so openshift, or what have you your canary deployments as well? So you can start doing things in DNS and pointing things to just subsets of users and running those experiments and comparing the two numbers and I think that's for me. That's really where the magic is and I'm not a front-end person, but I've seen front-end people do amazing things with blue area.

D

C

Just like targeted front pages and stuff like that and being able to actually look at statistically significant numbers and saying no actually, it looks like when the card looks this way. It might be ugly to you, but people get through the checkout faster and with a high rate of success. Yeah.

A

It's racing to me how the the the the look and feel can sometimes be the exact opposite outcome of what.

D

A

Would be right, like the wee-wee, we put a lot of effort into UX, the Red Hat. We see more effort coming every day and I know why? Now it's because of stuff like this right, um you mentioned like for Meg pictures like being loaded and instead of you know like optimized for the web type stuff.

A

A perfect example of something like this that you know I wish I had dated dog back in the day right when the Fukushima incident happened, the tsunami, the the nuclear meltdown, whatever I worked for a newspaper company and I was actually one of my like key things was managing the CDN so yeah, like all of a sudden, we start seeing this huge increase to one of our newspaper publications right, huge increase in traffic, just blowing the CDN table right, and it's like what is going on well turns out that that newspaper actually had somebody on assignment for a completely different topic.

A

When the event happened, they were sitting there with their. You know digital SLR, a satphone and an FTP connection and just uploading picture after picture after picture after picture into our CMS, and you know the journalists and editors of the newspaper loved it, so they just put them all on a web page.

A

So you had a couple dozen.

D

A

Files rendered down so that they actually fit on a web page but they're being served. Millions of time like this went like viral. The post actually went viral because it was like one of the very few people with you know live footage coming out of the environment at that time. So right like finding that out very quickly hitting our you know, you forgot to run the batch optimization process on these.

D

A

It's like it's saved tens of thousands of dollars in a day right, but that was just with basic billing alerts. Right, like we had crushed the billing threshold. If we would have had something first like this, we would have seen this huge increase and seen this huge anomaly happen and been like whoa. What's going on and could have started looking way early in the process, potentially saving so having this stuff sooner rather than later, is vitally important and I can personally attest to that through my s3 bucket was lately so yeah yeah.

C

I think there was something this chat about billing. So when I read, we can set up billing alerts so we're not like across management company or anything like that, but we do have access to this API, so you can certainly set up here, I think if you're already in a co-op or buddy, you might have something like that, like your AWS I think has you has like a billing thing built in so does Google.

C

So does this your a lot of people these days are in multi cloud and I realize that that's not it's, not everybody, but there are people doing it and setting up those billing alerts across multiples can be a pain. So that's that's. Certainly there I've done it myself to my trial. Accounts I've, certainly gone way over budget Oh, kubernetes and since I left running on a long weekend. It's generally my mo you set it up on Friday and you come back to say, look how that was yeah.

A

Yeah no Brian.

F

I had a question about how data dog is generally set up. It seems to me like wait.

D

F

Could set this up not well you'd set this up on probably on my development cluster but on multiple clusters, and is this going to show me traffic flowing between clusters for that kind of larger infrastructure view and could I have data dog on on gke, plus some data dog on openshift, possibly both installed via the operator? Would that make sense or or is there different installation paths, but still opportunity to view cross cluster traffic?

F

C

All says I should say so. I'm new to this role, like I've, only been a tech evangelist for six months. But what I'm learning to say is it depends I think this is like a common refrain so or maybe like yes and so. I understand, I've not done the multi-cloud tracing portion. That you're talking about specifically, but just to talk about how it works is when things get injustice when they first come in.

C

That's when we rewrite the header or add things to the header or passage on stream, so my understanding is as long as the calls going out also retain those headers between your multiple clusters. Then they'll retain them when they come in to the other one.

C

So it's not like it will overwrite or anything like that at the very least, you'll be able to see the traffic calling out from one cluster into another cluster if they're in the same cloud, no problem, oh so ever they're in the same app or what-have-you, although those infrastructure tags will allow them to really outside of them, though. So if you're going- let's say from kubernetes in AWS, to like a Red, Hat VM in Azure, for example, the only thing that's going to relate those two will be that header.

C

Basically, so my understanding is when you pass a tour from one to the other, that will actually retain the header and that's how you do it at the very least, your egress traffic, so your outgoing traffic will allow you to see that you're calling it you can see that trace therein in the last portion of that flame graph and, on the other end you'll be able to see that it came from the cluster.

C

Well, my understanding is that that actual header is what's going to relate the two, so that that's session that accession ID is what's going to relate to.

F

C

F

And and then are all of these metrics stored within each of those clusters, or do they stream out to data dog HQ where they're centrally queried yeah.

D

F

If they were centrally queried on your end, then I could see this being really easy for me to perhaps match up those those requests across clusters right. You.

C

Got it yeah, so it's it's a bit of both right so for us to handle the volume that we've got outgoing we buffer a little bit locally, which is good. So if something goes down, you'll be able to replay the buffer, but generally, what we're doing was we're streaming everything out as fast as possible to today to dog.

C

So we've got a catch phrase called like logging without limits, but it's the same for tracing I believe is tracing with those limits these days, but we basically want to take a hundred percent of what you're sending out and not sample so grab it across clouds and allow you the same thing like another catchphrase: I'd love to hear as a single pane of glass like putting everything on one page, I'm more like a like a single pane of glass for your usage kind of thing.

C

But if your use is let's look at all of my communities, clusters deployed locally or multiple clouds, or what have you? You can certainly put those all in one page, right, I. Think I can probably demo that here basically like what does that look like right. So how do I see all of the communities clusters together and we can? We can certainly allow you to do that from all of them. That's.

F

Not usually my my number one as an average developer, that's more like senior level, but it's definitely.

D

F

That I'm interested in hearing what your approach is, especially as more folks start thinking about multi cluster. If there's an easy way to map it all up before it's fully solved for the community right.

D

F

Start making sense of it today, then then this is, you know, have some clear use right.

C

So maybe I can steal if we've got some time just to steal an example from my past because of I've been at like a senior engineer at a pretty big company, so I think when I'm working with like dev team leads a lot of what they want to know is how is my app doing like a cross region and how is my app doing across environment? So if it's scene staging, they can come to me and say: hey it's fast and staging.

C

Why is this so slow in prod in Korea, like wow I, don't know, let's find out kind of thing. We certainly allowed that kind of view right so like what I'd normally hear is like I. Don't care about anybody, others service anybody else's service.

C

Where is mine kind of thing, so what I'd normally do is set them up like a like a dashboard quote-unquote, but like that, a page that percolates up all those metrics for them, so I think we kind of saw in an APM and I'll drill down into it again, if you don't mind, but basically the way we do, this is again tag.

C

So you can see the tags roll everything around here, but basically, when you go into like a service, you can then allow you to sort of drill down into each of them by just looking at like, like only look at this service for this region and I I sort of showed it in the communities one- and this is a can dashboard. So you're not going to see it here. What if we go back to the kubernetes one just like for one that you've made it on your own and I?

C

Don't I don't have an example for an app sorry, but basically these drop downs at the top here and it's a bit small, so I'll try to zoom in. But these allow us to say like let's look at just this one cluster or let's look at just this one deployment or this one namespace and you can imagine for an app like it would be. Let's say the environment is something like the let's look at just this one region or if it's at your environment, to something like staging.

C

Let's look at only the staging installs or let's look at it, test staging and prod for this one region. So there's different ways of slicing it, but I totally agree. You might not care about the infra. You might not care about the containers, you might not care about the controllers or any of that you just want to see like how's my app running, how many instances my app do I have.

C

Am I running up against like a memory ceiling that you've enforced on me, like I've, seen that before as well, people setting up quotas within communities for specific docker containers that are deployed in pods and making sure that you're not like at the limit or or showing like where you are in that limit?

C

That kind of stuff is really important as an app that if you want to make sure that, like the infrastructure serving your purpose or you know what the limits are, that you need to be working within in your purpose, we had a no doubt, for example, it would take about a gigabyte of memory and it was spective one megabyte of memory and it wouldn't take it right away.

C

He would take it over time, would take about a month but being able to see that crap go up and being able to show the historical track and be pretty handy as well. So that's a long answer, but like again it like a yes it, but it with it with the topping of it depends.

A

I'm going back through the questions here, um so okay, so I'm gonna, say the these name wrong, but cease and tampura sorry does data dog do you'll, contribute open source, wise to zip, can or open tracing or any other projects like that? Oh this.

C

Is that's a great question so yeah that's.

A

That's really the gist of what I'm trying to get out here. I think so I was, is.

C

Zipkin user prior to coming to data dog and there's a lot of work in that community for sure opens up can I think is specifically what we're talking about at that point. What happened is many people want to do their their own version of this, so so sort of two camps came out of that one of them was I, think it's open, metrics and then the other was open census and we basically we contributed to one I think it was all consensus, but we contributed sort of the tracing area of the agent for them.

C

What happened was there? There was like a I push to bring the community all together under the branding of open telemetry. So, basically again the answer is like yes, definitely tracing is important and what we do in the tracing world certainly does have an open source component, we're for open source champions, the digital most of what we do is open source. You can see the Cofer agent on github in cynical for like a lot of what we do for even just our like infrastructure architecture, that kind of stuff.

C

In terms of like how we set things up automatically also conserve got a pretty big github. Repo definitely do poke around. But what I'm trying to say here is that in open telemetry we are part of that community. We do contribute back and we're very interested in making sure that you know it keeps going forward.

C

So what we want to see is people actually involved in the tracing world, not just specific to us, but actually, you know reaping the benefits of the community here, because I think Zipkin was a big change in the way that we look at apps and it wasn't I wasn't pretty. It was a little bit hard to set up. There was a lot of moving parts to get it working, but once it was there I think that's where the magic happened.

C

Once you see these flame graphs of traces right across all your services, it it's a pretty it's a pretty powerful visualization of. What's actually going on your service, so yeah sweet. We definitely do contribute back to open telemetry, as well as the slew of other projects that one specifically is for tracing if you've not seen it definitely look it up. I think it's just open, telemetry, slash, open telemetry is is the main project and in the community you'll find people calling it hotels.

C

That's te l and we're certainly I'm involved for sure, but that's a great question I'm glad you asked so get a really good story around that one.

A

Awesome see the next question was like: can you feed these stats to some external service or like a elasticsearch or some other kind of thing?.

C

Right so um the way we work in in general is stats, D and there's ways of multiplexing stats, D. So let's say prior prior to getting us, you could take the same. It's for the most part. It's it's like vanilla, stats, D that we said we call it dog stats, D!

C

It's it's like a like a slight Fork of it, but for the most part, what we're sending from your apps is actually SATs T and the great news there is that yeah, their stats, D collectors for elastic search, you'll, see you saw em for graphite, oh that kind of stuff as well, but you do it like if you want to keep a collection on Prem as well as in the cloud, if you need to have that back up for any reason like if you felt like you needed to to keep some of that in your own or if you felt like our retention, wasn't good enough.

C

You need it like 10 20 years, and we give you a certain amount in the cloud and you want to make sure you've got it. You can certainly do that. So there are a few multiplexers out there for stats. Dean I can't really recommend any of them because I haven't used them, but I've talked to people at other companies that do you do it. They do work and there's there's different ways of setting up that. One cautionary note I have here is that there's a lot of metrics going out.

C

It's not the same, can't do it but be prepared for like a lot. Basically, there's there's a lot of signal that comes out of a typical coronated cluster and yes, you're being kind, listen I've done it before I set up 300, node elasticsearch cluster and seen it fail overnight. While I was like in the midst of needing a root canal. So, like I've got the horror stories there and I know it can be bad, but it's fun and it's it's definitely.

C

Good engineering like you've got the cycles if you've got the cycles internally to look at this or if you wanted to figure out. What's going on under the hood, definitely look at stats, T and what's happening there, because that's really the meat and potatoes I think that the same person who asked the question said like we predate some of this tracing stuff and basically the answer there is that we evolved with the community as it came through, like that's T's, completely open source. What we do is also there as well.

C

You can see the documentation on set like dog stats d compared to stats T, but it's it's important for us to actually see what's going on in the industry and serve our customers at the same time, while not falling too far behind, because it it changes really quickly. Right, like it feels like every week. There's a new way to monitor things or a new thing to monitor.

D

C

Possibly both so it's important for us to to be involved in the open source community and watch what's happening to make sure we're abreast of the changes so that we can help our customers, but yeah like learning about ceci is, is definitely really handy and if you wanted to like, we talked a little bit about monkey patching earlier, but really what that is is leveraging these sassy packages for those languages.

C

So, if you're looking at like an old language, what's a stats D look like for note that kind of thing it comes to tracing it's a little bit different, but in terms of the actual metrics themselves, yeah, it's mostly statsky cool.

A

So you could that, and you could totally like step D out to spark if you wanted, but lots of data be aware. Yeah spark expects a whole fun new ballgame, yeah, yeah, that's and I would like. I would like to have somebody come on talk about spark and like what you do with it with openshift right like in the future, but maybe for this channel, or this show you know, yeah I've been there done that yeah.

C

A

Was introduced to spark a few years ago well, many years ago now, I guess and I was just like wow, but the cool thing was. It was Alexander Rubin from Percona I think he works at AWS. Now, oh yeah.

D

A

He had like an entire like apache spark cluster like running in there like data center in their office, and he started doing his demo and you could actually hear the fan defensemen up, comm closet. Like start spinning, it's pretty funny.

C

Yeah, it's percs vikes, pretty interesting I! Guess that the main thing we're trying to say here is that, like that as a metrics come in and as you want to grab the signal from the noise from those metrics, it often means the infrastructure, and it often means like being prepared for that. It could be that you just want to first upset as well. I, certainly seen that like not grab everything, but just for this one out, yeah grab.

A

It grab it as some sample of sorts yeah exactly.

C

Or this one target area, are you right like us, like a sample rate or something like that, instead of a hundred percent, because that does become a little bit problematic as instead.

A

You go through.

C

A

Data transfers all that fun stuff yeah.

C

Engineering, ours was the other one that I've been dinged on like why your engineer is spending so much time on this thing internally, won't you just pay for a service. It.

A

Is yeah data dog? Is that ultimate build versus by question and that I think a lot of people have to answer for themselves and oftentimes. They realized that, like there aren't like monitoring, gurus anymore, like there used to be right like there's no one out there. That's like a Nagios expert right, you know, is that's a blast, not Android.

D

A

If, if you're uh yeah, that is a blotch one path, but if you're, if you're out there as like a monitoring person right, like that's a rare skill set, does that make sense right, yeah, please make sure you're being paid accordingly.

C

It definitely are still Nagios shops out there right for better for worse, you might have an area of it. Oh yeah.

A

C

A

It like don't I'm not, I'm not knocking Nagios and.

D

A

Skill set is an expertise that isn't like a huge focus right right now in the industry, for whatever reason, um but we we always say that monitoring is like one of the most important things. Yet we don't emphasize like the tooling to do it kind of deal so yeah that build versus my conversation is a huge one but I think I, don't know anyone, that's unhappy with data dog and when they decide to use a dog right like they.

C

A

Kind of a very, very heavy lift that has been taken from them right.

C

Yes, I will say, like I started out with Nike and cacti with a dog, okay and it's sort of silly, but what I was doing was keeping my negiah stuff locally and using data dog as sort of like that cloud. Backup, if you were just again, it's not not maybe the best use case, but it works for me excited swamp, Ram I was basically HIPPA like I was in a healthcare right area and I couldn't necessarily send everything to the cloud.

C

But what I wanted was like my events, my grass we do integrate with Nagios and cacti. There were older integrations, but they certainly do so if you've got those on crown-like. Don't love me. Let me laugh at that experience. Cuz I've certainly been there. I know that pain, writing pcre checks every day, all day yeah. But if you do need like a like a safety net kind of thing,.

F

Are there other integrations that we should consider as well like I know some openshift clusters already have ability for Prometheus? Is there? Would there be a good reason for me to take advantage of that, but also take advantage of data dog or they duplicate their.

C

Similar, um it's almost like a like a a thorough Venn diagram like there's, there's what we can do and then there's what Prometheus can do that being said, that that small circles, pretty big, you can do a lot with Prometheus.

C

In fact, if you're using like like a service mesh, for example, a lot of those meshes well actually have sinks in them for Prometheus already, Prometheus itself is actually like a a good way of setting up like a per cluster metric kind of thing, Multi cluster, maybe not so much and the scaling story around Prometheus itself like there are a couple projects at the LP.

C

Do it generally not something that like it's, it's meant to have like one Prometheus and then, if you want to scale, then you figure out how the multiplex, at that kind of thing what we provide there is similar to like the Negi O's. We can aggregate from multiple previous installs or if it's just the prometheus, install that you've already got. We can certainly grab data from there. So if you've already got Prometheus like there's, no no real step to like our integration in communities is, is set to work with Prometheus.

C

Well grab those from you there's a couple: config changes you need to do, but they're they're, basically there as like a and you say it's like a default config, so the endpoints already define is just common. Today, you'll uncomment it and you'll start grabbing from Prometheus for you Prometheus itself like, but it it's sort of caught on quite a bit right. It is really popular in the open-source community. So the onus is on us to integrate with that right as opposed to just ignoring it. Not it like. Definitely, if you're using prometheus like.

C

Why would we not help you get those metrics into the clouds we? We certainly do integrate to it, but yeah like a I've, dealt with this to do clusters and a lot of the info that you get from any CEO like sorry, initio cougar and is to enable kubernetes cluster lots of words in there yeah, but they already send as the metrics to that sync already. So why not just grab from there right? Why? Why set up a whole new path, kind of thing, yeah.

F

Awesome, that's really cool yeah.

C

Sorry I didn't want this to be a sales pitch, but yeah. We definitely do that degree with no.

F

I wasn't aware of that and I was kind of hesitant to ask that question, because.

D

C

F

Like here's a competitor, what do you have to say about it? No.

C

No not at all, really good.

F

To know that I could just continue to use Prometheus when it makes sense and then wire up with minimal effort and provide visibility to other teams or whatever's necessary. You.

C

Got it yeah so so, for me, at least, prometheus is kind of like a like a metrics database like it's a really good way of storing metrics or in some cases like the meadow did about traces, but maybe not necessarily like a way to relate all of those very easily. The UI I can that there are front ends for a theist for a reason right.

C

So some of the value that we provide there is just a way to correlate between all those metrics or aggregate, or look at your percentiles for those metrics, which Prometheus also has just in a way of looking at like multiple clusters or multiple installs or multiple different services in just a different way. A.

A

Few questions here from our Carlos Santana and there's a typo and one of them that might be offensive to people so I apologize, yeah.

C

I think you mind: yeah.

A

Like our I think.

C

A

Not having a fun day today, with moderation, I'll just leave it like leave it at that. The the first question you asked about the pod scraping Carlos. Can you ask that please, mmm you know I'm, not quite sure what you mean there, but for service ryan mclane. Would what would we do for service metrics there? You can't really scrape you know you kind of have to omit those. What do you recommend? Yes,.

C

In this case, I must admit: I've I've not done server listing kubernetes I know that's, maybe a full pot I've done service. I've done grenades, I've not done one inside the other. That means that I can sort talk about how that works. So in general, when you're loading your libraries for whatever application you're running so let's say it's Ruby or PHP or Python, or what have you that's when you'd actually load in sort of like the data log library that would emit out to either like a public endpoint or if its internal?

C

We can talk about like that dog stats de sort of workflow and you would actually send it into there. I've not done server listing meeting metrics to Prometheus again because I've just not launched service in kubernetes. That being said, if it already does that it would be the same way now in terms of tagging, the the way it's sent out for us like if you've tagged it in Prometheus, we'll grab that tag. Basically, if you have not them, we won't have the tag to grab.

C

What we will do, though, is things like the cluster of the region. The availability zone, all that stuff where Prometheus is launched, will get, but if there's anything specific to your function like the function name, no problem, that's metadata that we'll get if you need to have like a function name and a version pair together like if it's function, joel's awesome, function, v1 kind of thing.

C

If we want to like tag the v1 and do that string concatenation with the tags, that's something that you probably want to do, as you admit, add to Prometheus, so that we can scrape it from Prometheus. For you there you go it's a long answer, but I think that's how it'll work yeah now, there's a what are the what's the state of serverless in kubernetes, cuz, again, I probably should know I I know there was Oakland whisk at one point, a.

A

Native Canadian, it was.

C

A

The world is moving right now. I know that we have a lot of Canada's a lot of effort behind Canada right now we being Red Hat but I- think in general Canada this kind of getting the the lion's share of serverless stuff on kubernetes right, like oh, that makes sense. Yeah.

C

I think that's what we make sense. I I just have not used it at this point. I I did play around with Oakland West but honestly didn't get correct same.

A

Likewise yeah was it I think it was 2017. I was sitting next to burst, shutter at all three Lawson and he's like hey yeah man, great great, to have you here, blah blah whole was before I worked at Red Hat he's like yeah we're looking into open west for a bunch of stuff and I start looking at it like. While he was talking, I was like you're.

D

A

Get pretty far with it, you know so: yeah 2016 yeah yeah, there you go yeah. So it's it's one of those things where it's like. We ship an open-source ARELIS. Yes, we do. We have a product for OpenShift server list. Oh sorry,.

C

I, really it's based on K native thanks. So.

A

Yeah we weren't talking about product specifically but yeah, K native the projects and then product wise, there's open ship service which is based on K native, so yeah. Thank you. Langdon.

C

For dropping the links from chat- oh, this is great, so I I had a team like I helped deploy as your functions has a library that you can deploy into a Cass but I honestly sort of turned a blind eye to it. It ran and it seemed to work, but I didn't I, didn't really know what's going on. It was just like that, like a schedule, tasks kind of thing, if you've ever done, those in kubernetes like once a week run this task or any kind of thing.

C

The service seemed to be a good fit there, but we didn't other than it running. Reading revenue metrics for me.

A

Cool all right, I, don't see any more questions or pass time here and if you, if you do, have any questions, where do people find you Ryan.

C

Yeah, so on one twitter is probably the easiest way either that or I'm in a bunch of the slacks, but I'm generally either myself on twitter. I I miss getting my my vanity username swamp ryan, underscore mclean as opposed to ryan, mclean and poor thing yeah. I know you can't win them all right, but that's something into it. Otherwise, I'm just Ryan macLane, all in word, I'm I'm out there. Let's see if I can stop this zoom here but yeah. Basically you can. You can find me anywhere. Ask me any questions.

C

We, we didn't actually get to Joel's question, but that that the main thing I was gonna say it's just like if you're in node any want a to dog, we've got talks for their great docs, its one-liner, its its import data. Dog, basically, is what does all the magic and everything is done aside from the environmental variable, so apologize for not getting that we got into a bit of an SRE rant, which is a favorite topic of money. Yeah.

A

Sorry I might have been my fault too. Certainly a guilty party here all right. Well, thank you, Ryan and Ryan angel' for joining us today. I really appreciate you coming on Ryan, McLean and Ryan J. Obviously, angel can't forget role he's what made this all possible. So thank you very much. Everybody join us here at 2:00 p.m.

A

Eastern, 1800 UTC, we're gonna have one of my favorite interns here at Red Hat this year, Cedric he's gonna be doing like his first six weeks with open shifts so Jason Toby's is gonna, come on on with Cedric and kind of walking through what they've been up to during his internship. So it should be pretty fun and I'm looking forward to it for sure so yeah until then we'll catch you next time thanks everybody.