Red Hat OpenShift Ask an OpenShift Admin | Red Hat Livestreaming, 15 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ask an OpenShift Admin (E93) | Openshift Logging and Observability

Description

Elasticsearch? Loki? Fluentd? Vector? What does it all mean?!

This week we are joined by Jamie Parker, product manager for OpenShift logging, to discuss the changes which have been happening with the logging service.

Want to learn more?

Try Red Hat OpenShift: https://red.ht/3uVxkya

Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Jonathan Rickard https://twitter.com/jrocktx1
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_c...

About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3rYz5ZS
OpenShift on GitHub https://github.com/openshift/

#RedHat #OpenShift #Kubernetes #ContainerPlatform

A

A

Let's get into the big topic of Open Source.

A

Ecosystem really boomed.

B

Hello and welcome to this week's ask an openshift admin office hours live stream. I am your co-host Andrew Sullivan joined, as always by my co-host, yeah we're co-hosts.

A

uh Johnny Richard.

B

A

Up buddy how's it going good to see you again, yeah.

B

You know it's only been quite literally a half a world since I saw you I, know yeah. It was good to see you get to hang out in person for the first time in ages. uh What I think since 2019 yeah it just so happened that we had to go all the way to Singapore to do it so yeah, it's good to good good to get all that done. You know nothing like spending. I think it was 26 or 27 hours to get home.

A

Yeah time zones are hard. Oh my God, oh my God.

B

A

Think for me, with like the layovers and everything was like a 33 hour jaunt to get there and then yeah you saw me there. I couldn't sleep at all, so it was a nightmare for me. Yeah yeah.

B

uh Well, we are joined today by Jamie Parker, who is on the product management team for openshift logging and observability, so welcome to the stream Jamie.

C

Thank you. Thank you for having me.

B

Yeah, this is one that uh you know we we said before we told you that um there's been a lot of questions. There's been a lot of interest in openshift logging and the stuff that's going on there. You know the the move uh from elasticsearch to Loki the move from fluent D to Vector, so uh I I hope our community has come prepared or let me rephrase that I hope they throw tomatoes and not rocks. Maybe.

C

B

So, but no it's uh I think this will be a great one, I'm looking forward to it, there's a ton of stuff that, uh quite honestly, I just don't know about it. Like I've, been aware of these changes, but uh aside from the surface level like yeah, we're changing things um to improve um I, don't really know what all that means, and I'm I'm also particularly excited about the and observability in our title here, um because observability is a bigger thing right, it's bigger.

A

Than just logging so.

B

uh Frozen Tomatoes- that's not very nice, Christian, all right, uh so um yeah we'll get to that in just a minute. I think um you know. Normally we do our top of Mind topics um and quite honestly, Johnny and I have both been gone for two weeks, so we just haven't kept up with a lot of things, but I wanted to take an opportunity here to mention a couple of things.

B

um So one and first I did look and see that the upgrade path to openshift 4.12 is still in fast, so it has not been promoted to stable. Yet that's normal. It's perfectly expected I. Remember was it two weeks three weeks ago, Johnny when we had Scott on Scott, was saying that normally it takes somewhere between 45 and 60 days.

B

So we're we're not to that point yet so don't be worried if you don't see the upgrade path and also I'll point out that the eus to EOS upgrade path that doesn't become available until it's promoted to stable 4.11 to 4.12 upgrades are promoted as stable, so keep an eye on that other than that. I wanted to take a moment and really say.

B

Thank you to our audience, especially all the red Hatters who watch, because both Johnny and I were absolutely Blown Away by the number of folks who came up to us when we were, you know going around and, and you know doing the thing and they just came up to say hi and say thanks and talk about. You know the stuff that we've done on the stream and really I I can't. Thank you all enough, like it's really really awesome, not just to get that positive reinforcement, positive feedback, but to know you're listening and you care.

B

That's. Why we're here yeah.

A

That was awesome, especially like in Singapore, when we had like we're taking pictures with people. I thought that was it was awesome. It was amazing.

B

Yeah yeah- and you know same thing in in Dublin and I- wish that one of us could have gone to one of the Americas, uh although the folks down in Buenos Aires, that's all happening right now. That's happening this week, so I think they're doing the keynote right about now so um yeah, it's just been.

B

It was really great and it was a lot of fun and um you know please don't hesitate to reach out um whether you're a red Hatter, whether you're a customer, a partner uh whether you're, just someone who's passing through, don't ever hesitate to reach out to us. You know we're always happy to do whatever we can to help and talk about anything. That's interesting uh and all that so yeah.

A

uh Yeah for sure and I was in Dallas and uh the weather. You know it was just so crazy and um it made it. It made it challenging to meet people, but like the the little bit of getting out and interacting that I did. I was able to like catch up and we got some positive feedback from there. So um when I told them that we're doing the logging uh talk when we got back there's the one guy that actually asked for it like directly was like yes, it's about time. I can't wait.

A

So it's a good way to come.

B

Back uh Khalid I, don't know if Network observability, I so I, don't think we're gonna go into details. I feel.

C

Like I can't talk about that, okay.

B

Okay, I didn't know if that's something that you would want to talk about or if we should go and poke at you know, Mark, Curry or or he's.

C

Mark Curry is going to be the expert on it, but I can at least cover a bit of it.

B

Awesome because I know that there's been some questions about that as well, um so yeah uh so kind of moving on from there. um You know for anybody. Who's watching us live, don't hesitate to reach out in chat kind of whatever platform you're on whether it's YouTube, whether it's twitch et cetera, all of the chat gets aggregated into our interface here. So we'll do our best to answer those questions as they come up.

B

If you're not watching us live if you're, watching, recording or listening to a recording or anything, you can still reach out to us and we'll do our best to to respond to get answers to include those in future streams. So for me you can reach out to me at andrew.sullivan.redhat.com sorry, andrew.sullivan.redhat.com.

B

Or at practical Andrew across all the various social media, platforms, uh Twitter and Reddit, and so on and so forth, um I feel like I, should get a mastodon account now I, don't know about that time! Yeah! That's a it's! It's a whole! um You know uh uh spider's Nest, that I'm not sure I want to dig into yet.

B

uh You can also reach out to Johnny uh j-o-n-n-y redhat.com, uh no h in there or at Jrock tx1 across all the various social media platforms. Another great way to get in touch with us. So for internal folks you know the normal. You know chat and email methods, but for anyone you can reach out to us via the kubernetes slack, so kubernetes slack has a openshift users, Channel or yeah they're called channels on on slack I, gotta. Think of all the different chat platforms and all the different terms that they use right.

B

So there is a uh a slack channel that you can reach out to us on that we try and uh haunt frequently um is there an IRC Channel um I, don't I Pro, probably um honestly I, so red hat still uses IRC engineering especially uses IRC for a lot of things. um I I am up to four chat platforms and I. Just I don't have the bandwidth to add another one. So I would um request.

B

um Muellera apologies for butchering your your name. There slack is probably the easiest one if you want to do the real-time interaction thing plus, there's a whole Community here, it's like 5 000 people on that slack Channel. Now uh the link is just up above. um So it's that kubernetes.slack.com link that was posted by um you're on YouTube, so it'll be the open shift to the red hat user.

A

I'm posting it again, but up that finger then I typo, the name. How dare you I'm I'm, the worst I thought we had a shortcut for that we do but I can't I couldn't remember it naturally.

B

But it has been three weeks. It has been three weeks um so yeah. So yes, today's session, today's stream is about openshift blogging and observability. This is a office hours, uh so office hours means that you can ask us about anything and everything. That's on your mind, um Johnny and I will do our best to answer those questions, um probably in chat.

B

uh You know we'll Bubble Up questions and talk about them occasionally, but we really want to give Jamie enough time to talk about all the stuff that she's got going on, because, like I said, this has been a super popular topic and I want to I want to make sure that you all have plenty of time to ask questions so that we can. uh We can get through that. So all right, I think I uh did I miss anything. Johnny did I ramble through all the normal stuff, yeah.

A

I think we're good.

B

All right uh so Jamie I will uh I will toss it over to you.

C

B

Know you had a few slides to kind of frame the conversation and set some stuff up so to the audience feel free. um You know, let's, let's keep interacting in chat and all that Johnny and I will certainly um talk through that and we'll ask questions as they come up with Jamie.

B

um But you know we can definitely really go into conversation um afterwards and I do have a cluster here. If we uh want to show anything off.

C

Okay, see all right looks like that came up um so I do want to talk about openshift vlogging course I'm, the the PM for logging, uh but I also want to talk about observability. uh What a lot of people don't realize is that we do have an observability platform within openshift, so I want to talk and touch a little bit on all of that today and um talk about network observability is included in there.

C

So we have an observability strategy in the observability program team, and so we want to develop one unified, consistent, simplified footprint that can work across all of the environments, so public cloud, on-prem and Edge it'll be the same platform for observability for all of those, and to do that, we have we're trying to break down the silos of logging, the silos of metrics, the silos of tracing and we're trying to turn it into Concepts. So we have several Concepts here for data collection, so we can do that with metrics views and Prometheus.

C

We can do that with logs we're using vector um data analytics so querying metrics uh filtering logs, we'll go back a little bit more into detail in that. In a minute um visualization, so we are trying to build an out of the box observability platform experience within the openshift web console natively.

B

And and to be clear, Jamie, this is it's an overall strategy for how to bring visibility, how to bring insight into the um multiple pools, multiple sets of data. So that's.

C

B

Networking metrics is a part of that logging data is a part of that. um You know and a bunch of other stuff I'm.

C

Sure, yes and cluster metrics um lots of lots of different metrics on your your openshift platform. So you know day two if you log into openshift- and you have all of this available to you so.

A

C

Delivery um normalizing the data with observability operator and then also um transporting logs, wherever you may need them with vector um and then storage, so we can store metrics with Prometheus store Logs with Loki. So this and then we have third parties Integrations with all of these. So it's a whole platform within openshift that exists already foreign just to talk about what is happening now with openshift 412 um logging, five six, so you can have topology spreads with Prometheus and alert manager. So you can spread your different alerts across different areas.

C

Depending on how your clusters are set up, um you can create alerting rules based on those platform metrics. We have run book URLs in the alerting uis. So if you have a run book, you can go ahead and run that to get some normalized alerting and then we're always working on improving our whole UI experience. So uh now we've improved the metric selection, so you can select metrics and search for those easier within the UI um for logging. Specifically, we are now logging to Splunk and uh to Google Cloud.

C

um We had a lot of customer requests for cluster IDs to be added to log records so that people can search through those aggregate those a little bit more easily.

C

um We're now fully released a vector in uh in place fluent d, um and then we have additional log exploration options in the UI and I have a slide that shows those in a minute. So you can search through your logs a lot better um and then we're starting to introduce open Telemetry. So we have a tech preview of open Telemetry for tracing.

C

um So this is just what we have going on now, we're always building on all of this. When uh you know we always are looking for feedback from you know: users, customers, um internal people, because we want to make this a um a really well thought out platform.

C

And just to talk about all of the open source projects that we're using for this platform, uh we have Kafka Prometheus, Tempo, Thanos, of course, Loki and Vector for logging, um Jaeger and open Telemetry, and we do contribute to all of these projects so across the observability platform team, we're actively contributing back into all of these open source projects.

B

Jamie I'm not going to ask how we avoid copyright issues with these Community names.

C

um That is a good question. um Actually I don't know the answer to that question.

B

I'm sure I don't want to know I'm just going to let it be.

C

Yeah uh but now it says, um since the request was openshift, logging I can go now deeply into openshift love. You.

A

Hey and real, quick before you jump in, um we have one question from uh Khalid and it's uh so when you upgraded your ocp right, I'm going from 411 to 412., um will it automatically switch from the traditional openership logging over to Loki, or is that something that we would have to upgrade to no.

C

You would need to upgrade well openshift, so Logan and Vector are they exist. But if you're currently using the previous elastic stack, you would need to do a manual switch over to vector and lookie. Okay.

A

C

Automatic thing.

A

All right so clean, hopefully that answers your question. uh If not go ahead and ask uh you know more clarifying we'll get it back to Jamie all right thanks, Jamie sure.

C

um So initially our opening logging stack was elastic fluent Cabana. um This is a fairly standard configuration so elasticsearch for using storage and searching uh fluent for collecting and transporting logs and then Cabana for visualizing those logs.

C

um So after running this for a while and working with it, uh we made the decision to move to Loki and Victor. So this is our current logging spec, and what we found is that for our needs, as a logging, team, uh vector and Loki uh performed better, so the indexing in Loki is different.

C

Instead of indexing, individual lines of logs um Loki will do a log labels, and so it's a little bit of a faster performance when you're trying to parse through those logs uh Vector, it has a really good performance in logs per second, we found that to be more performant for our needs.

C

um It's there's a little bit more better support for pod labels.

C

um Look the nice thing about Loki is that it allows multiple tenants into one storage instance so that we found that to be very helpful, and then there are a lot of uh native plugins with vector and Loki that we've been able to utilize. So this stack we have found, performs better for us overall and then also lets us go into more deeply into. The observability needs that we have trying to support the observability platform yeah.

A

That's awesome and it sounds like there's a lot more integration too, with third-party components so.

C

A

I, don't want to use this natively or, if I don't want to use this exclusively I can still interact with those logging mechanisms that I had before. So yes,.

C

And Vector is very um neutral to vendors, so it's really easy to uh or easier to develop the different forwarding outputs according to whatever customer needs. Oh.

A

C

um So right now for Vector we have um the different collectors and forwarders uh these are configured as of logging, five, six! um So Loki! If you want to keep your logs in cluster, so you can use Loki, that's uh you can install the looky operator and use it so Vector would collect the logs and then send them to Loki.

C

um There's a couple of advantages to doing that. But then there are users who need to send them to other places. So right now we have um collectors that are configured to send logs um off cluster to stackdriver through Google. um You can send your logs to cloudwatch. If that's what you're using and then many many customers are using Splunk. So we have that configured to send those logs to Splunk. So um you know, as as we listen to what customers need.

C

We create more of these forwarders and uh just trying to make this a more usable platform for everyone.

A

So I see you have gcp and the AWS stuff are: do you have Integrations for that? I, don't know what I can't remember what it's called for. Azure.

C

But yeah the uh cloud or log analytics: okay,.

A

C

All right, that's coming in future. We don't have that yet, but that'll be in the future. Iteration.

A

Okay yeah, because that was a pain back in the day like with like the old, uh the old sex, so.

C

A

C

We're we're still building. We have a lot of things in Flight that we're building out. So um you know if anyone has anything that they particularly need. um You know that's the good time to reach out to me and we can take that back and evaluate.

B

Awesome, thank you and I was in the middle of a coughing fit so I. Couldn't I couldn't comment at the time but I. Remember there being some issues with fluent D, um effectively dropping messages once there was I, don't know if there was a consistent threshold, but at some point it would basically uh overwhelmed isn't the right term, but it would reach a point where it started: dropping messages which is, of course, an issue. So I know that was one of the major factors for looking for something else and I think ultimately deciding on Vector.

B

But I don't want to speak for you.

C

The your vector we found has um Vector is a lot more scalable Vector. Is it uses CPU a lot more um flexibly, so if it gets the more logs that it ingests the more it's able to actually scale out, use the CPU appropriately and then get those logs where they need to go. So that was one of our big factors in uh deciding objective.

B

Yeah makes sense: uh is it a good idea to forward logs to be realized from Khalid.

C

You can't speak to whether it's a good idea, I mean that's that depends on whatever the customers need, but um uh at this moment we don't have a native integration for that, but we can, you know we're. Certainly uh if you have an rfe we're willing to take a look at it.

B

I mean I would say you know if you're using vrealize log, Insight um or whatever they're calling it now uh as your log aggregation log analytics. You know interface, then sure, there's nothing wrong with doing that.

B

um You know it is I, don't know if it's a competitor, but it certainly appear in the broader observability. You know: spectrum of things compared with the openshift observability um I'm, going to say portfolio, but collection of capabilities, um so yeah. If that's what you're using there's nothing wrong with doing that. In my opinion,.

C

um It's it's: whatever users need every company, every user has different needs, so whatever they need. Yeah.

B

It's like Splunk right if, if Splunk is your standard for where you capture logs and you use that to do analysis, then great there's, nothing, no reason not to do that and same thing with whatever other solution you may be using yep.

C

So, on the Loki side, I'm switching to Loki for log storage, there's a couple of advantages here, one of the biggest ones that we found is that you can use log ql for querying these logs. So just like you use promql for Prometheus metrics log ql is very similar, and so it's a really easy shift into querying for logs for many people and that's been a a very positive thing.

C

The way that Loki uses memory is a lot more efficient, so it labels the logs and places them into chunks. So when it goes to index all of those logs, it actually indexes the chunks and the labels. So when you're going to search the searching ones are being a lot faster because of that um the multi-tenant storage is very again very useful. You can write all of your logs to the same tenant uh from multiple Loki instances, and then there is native data isolation within that, so your security is preserved.

C

So the uh that's been very helpful.

C

We've added cluster ID as of logging, five, six, so that's makes it easier to sort through those logs aggregate those logs and then, when you use Loki within openshift, as the openshift log store, you get a lot of log analytics that are native within the console UI and just to show you a little bit of what that looks like. So we have the observability um view here within the developer console, so you can aggregate these logs and you can search for them by namespace, there's an automatic filter for that. So you can filter by namespace.

C

You can filter by pod and you can filter by container and then drill that down so that you can search more efficiently. So these are some of the things that we're trying to do within the observability platform.

B

Jamie, do you know um and I've seen this question come up a few times? Do we I, don't think we document I assume that Upstream is documented, like resource requirements around like storing logs? How much you know capacity gigabytes it. It takes how much performance you know. High obso latency is required for those types of things we.

C

Have some um we have some recommendations on sizing, especially when you get into um like AWS instant sizing? um It's really hard to. We do have some IELTS recommendations as well in terms of just raw storage. You know that's a harder recommendation, because every um every usage is different yeah, but uh we do have some uh documented iops and um instance, type recommendations.

B

Okay, yeah I know a lot of people um used to ask like with elastic elastic. You know how much capacity should I give it. Well, how much logs are your applications generating? um You know? Yes,.

C

B

There's no hard or fast rule, um you know, 10 pods doesn't mean 10 gigabytes, 10 pods might be in one gigabyte or it might be in 10 terabytes. um Exactly.

C

B

But yeah I didn't know um if we had any any other kind of recommendations, because I haven't honestly haven't looked since the 410 days, so we.

C

Do in the um openshift 412 um documentation in the logging section, there are some some recommendations on uh iops and instance: types.

B

Got it? Thank you sure. um Looking through chat here, I see there's a couple of questions. uh I'm, probably going to get these out of order, so uh hi K, again apologies for butchering names. Can you please elaborate more on multi-tenancy with Loki, so.

C

Multi-Tenancy, all of the logi instances can write to the same um log instance. So you can have one log store and lucky is uh built to write to object storage. So everything can funnel into that same object, storage, piece and then, depending on the permissions for namespaces.

A

C

Will be the logs that you have access to within that um within that storage.

B

So does that fit in with the um sort of openshift platform plus ACM ACS uh Hub type model, where you can have a central low key that other clusters use or you.

C

That is uh I, don't know the full answer to that one. um Maybe we haven't really tested that out completely and we are still working on some aspects of multi-cluster. That's on our roadmap for future. So um I'm not sort of the answer, though,.

A

Yeah, that's great wow. Sorry, words are hard. uh One of my one of my customers a long time ago, was like real big on Multi. uh The multi-cluster uh logging, so I mean I. Think that that's gonna be an outstanding feature, especially like I mean even I think that they would have been happy with the multi-tenant aspect of it, but uh to go multi-cluster like the way that Andrew was describing would be. uh That would have been ideal and probably would have been a crayon sign for them. That's.

C

A yeah, that's it that's a um it's! A big ask. It is actually on uh I'm gonna talk about it in a second, but uh that is one of our things that we are working towards.

B

Yeah my mind is full of things around like hosted control planes and all this other stuff that I'm sure you're, you're well aware of and well ahead of me on so.

C

Yeah, if you, um if you think about it in visual terms, you're taking a flat, topology and you're turning it into a cube, and it just you know, it starts to spin and it's it's.

A

C

It's a big lift.

B

Yeah well, while we're on the theme of uh you know, marvel you know, Tesseract and all that stuff.

C

B

uh Let's see uh so Oscar um I see your question here about fuse um and spring boot. I'm, not sure if you're, asking specifically about those things because um seems like openshift deployments, could be easier and have more features. Are you asking about like openshift clusters versus kubernetes clusters, um which or or is there something else there? If you don't mind, just clarify there, um oh I see you commented down below. um Is there a question for you Jamie? Is there a way to Archive Vlogs to external storage like NFS.

C

There you would, you would need to depend on the your storage capabilities, so um you know whatever store that you're using your underlying storage, you could. You would have to do um ship your logs through whatever's, for that storage method, advises.

B

Okay and I I Kelly had mentioned some. If you don't utilize, odf storage here I would assume that the nuba aspect of odf, which can act as an object Gateway, can kind of proxy that data to wherever you want it to be at yes,.

C

And um you can also use odf, uh multi-cluster, Gateway and uh then odf would act as object storage for you.

B

Got it uh let's see Oscar, can you help me understand how many apps can run on a basic openshift subscription? um So, yes, openshift is entitled on a core basis, so one entitlement is two physical cores or four hyper threaded cores um and the number of apps is going to depend on your application right. If, if your application isn't using a lot of CPU, then you can put a lot of you know a lot of applications in there.

B

If your application uses a lot of CPU, then maybe it's fewer, so it kind of there's a number of factors that play into that. One thing to note: so if, if you go to try.openshift.com.

B

You'll get a link to a bunch of resources there, where you can get started with openshift openshift also qualifies for the developer entitlements. So if you go to developer.redhat.com, you register for an account or if you already have one, it's a free account and you get up to 16 entitlements of openshifts that you can use to deploy uh a or as many clusters as you'd like again 16 entitlements, and it's free. So long as it is non-production single user use cases so a great way to get Hands-On with openshift.

B

um If I has uh yeah I'm wondering so, Sapphire um is super new I want to say it's Johnny I, don't keep up with Hardware I, don't know if you do it's it, it's less than a year old, so I I'm wondering if um it's just a kernel, you know kernel Drive. The kernel in 4.10 doesn't have the drivers but I'm curious. Why, like 412, wouldn't wouldn't be there um yeah I'm I'm interested to see how core OS layering might affect that.

B

If you're able to add the driver in uh Rick, uh you could run a two course subscription um it. Wouldn't you wouldn't be able to do a whole lot, uh so remember control, plane, nodes, you don't have to have entitlements for uh and infra nodes you don't have to have entitlements for so you can take things like um you know the Ingress controller, pods and all the other stuff and put that either on the control plane or on infra nodes, and then basically those two cores would just be the worker node.

B

um Although I I think the worker node the actual kublet and stuff like that, does reserve like half a CPU so yeah, it would be one and a half cores worth of uh application resources that you'd have available.

B

B

Hello, oh Khalid, uh how to generate alerts based on logs. That was an interesting question that came up um actually in internally the other day as well. So Jimmy I, don't know if you have something off the top of your head to help.

C

I, do that's uh that's actually on our roadmap to be released in um logging, five, seven! So we're targeting logging, five seven for that capability. Okay,.

B

Yeah I was looking to see if it was possible um just flipping through documentation, proverbially, flipping scrolling, I guess um through documentation and I didn't see anything that stood out to me. So no.

C

That's a big customer request and we do have that um we're actively working on that right now.

B

Okay and is does locking, have a similar release Cadence as openshift is.

C

It yes, so which we tried to release uh relatively the same um for the non-eus releases, um so we're looking at May for logging, five, seven, um which follows openshift tentatively, releasing in uh April and then for the EOS releases. We try to release within two weeks.

B

C

Cool, so when we follow a rough Cadence.

B

Yeah logging, like several other aspects, service mesh, openshift, virtualization Etc, it's a what do. We call them an a layered offering, so there is not an additional entitlement, but the release cycle is decoupled from the core open shifts, but I think all of the layered offerings generally try and follow openshift releases pretty closely yeah.

C

That's a um one of the one of our big feedbacks has been.

A

C

Know please try to follow the EOS releases, so we do try.

B

Yeah and nothing against the uh the odf folks, but they seem to be the ones who often lag the most uh when it comes to following the uh openshift releases. But that's fine. It's a complex system that they got going on there, so I get it especially um Johnny. One of the things that you I think you mentioned bringing up and talking about is the uh toppo lvm operator and.

A

B

Being able to utilize with single node openshift, the local disks through a CSI provisioner, which is really cool, that's.

A

B

Mira uh does open just support, running 32-bit applications uh yeah it's as far as I know, um 32-bit apps can run on a 64-bit processor. There's nothing wrong with that. uh Does it support multi-arch nodes? uh Yes, you can have I believe with 412. We support a heterogeneous, yes heterogeneous clusters, maybe that was even in four or 11 when we added that, because I think it was four or eight or four nine. We added support for arm or no 410 was I.

B

Think one of NGA we added support for arm, so you could have a cluster that was entirely Arm based, but you couldn't have the control, plane, B arm and compute nodes, bx86 or something like that, but that is as far as I know now. Now supported.

B

um So a question for you: Jamie: does the logging stack use more storage if the number of worker nodes increases.

C

I say yes, because you're going to receive a certain amount of um like kubernetes logs, so you would that would that portion would increase, but as for the rest of it, um it would probably just depend on the different applications that you're running you know if you're increasing the number of applications, it does make sense that log storage would increase as well. So.

B

Yeah, that's what I was thinking the just just adding a node. If it's not doing anything, will add a little bit of additional right.

C

B

Doing its thing and all that other stuff and it'll generate logs that'll get sent up, but it really it's dependent on the application, so in theory you're adding more nodes because you need to scale the applications more. So there's kind of this natural assumption that yeah you would. You would get more logs because you're hosting more applications. Yes,.

C

So that it's it's the very frustrating technical answer of yes, but it depends yeah.

B

There's so many things do these.

A

Days but the I mean the good part too. Is that it's tunable right? So you can you can tune how long you store stuff locally before it gets exported out or whatever so I mean like um you know, if it's something that like, if storage or persistent storage for an extended amount of time, is an issue, then you can tune your operator to export that out uh to your your long-term storage. A lot sooner.

C

Correct and if you're, depending on your S3 Source, you know it grows along with your needs, so.

B

um Mira x86 supports 32-bit, with no issues on all openshift versions. uh Yes, I, don't know if we have documentation on that, it would probably be in the Rel documentation. Actually, so it that's less an open shift thing and more uh container thing and the containers are Rel, because core OS is Rel, just a a flavor of Rel that is specific to openshift and for hosting you know, being kubernetes node hosting containers so somewhere in the Rel documentation, I'm sure that they have that, but I don't have that off.

B

The top of my head, we'll see if we can dig it up.

B

uh Rick learning with Loki is certainly super cool, because the query language is similar to prom. Ql I'll, admit that uh I am not good at prom. Ql I have not spent enough time with primeql to to be able to intelligently um and reliably get what I need out of it. But um you know so somewhere in my backlog of things. To learn about is prom ql.

B

uh Just kind of looking through chat here, making sure I didn't miss anything Johnny I know you sometimes keep notes of things we haven't addressed. Yet no.

A

We're we're actually tracking things pretty well, it's actually starting to look up the core Westing. It's okay sounds like the multi-lib uh libraries. Then the 32-bit things should be should not be an issue, but I would have to look in the in the core OS build and see. If they actually do, multi-lab I don't see why they wouldn't.

B

So um so, yeah, sorry, sorry to uh kind of distract you or go down a rabbit hole there. Jamie.

C

No no worries good questions.

C

So um it was a good actually actually good place because uh we've talked about the past of logging. We've talked about the current logging, and uh this is the part where we talk about what's coming up next, so this these are our upcoming items. So not all of these have um Hard dates, because they're all being scoped and worked right now, but for Vector we have HTTP forwarding so forwarding to a uh HTTP Target, which is a a big request.

C

Syslog forwarding another big request, so those are coming up soon and I mentioned multi-clusters, so multi-cluster collection and storage is on our roadmap and again that is a big.

C

It's a big way of thinking about it, like, like I, said earlier, you're, going from a flat topology to a cube topology, so it's fairly complex, but uh it is in the works um Locate. The log base of learning like we've already talked about, was a big request and then the Zone aware replication, so that is coming in upcoming versions and then we're always adding more um log viewing and aggregation enhancements within the console, UI and we're working on correlation. So that is something that we're working on.

C

In correlation engine at the moment to help with not only logs but also correlating um signals across all of the observability platform, so that was uh we're really looking forward to that.

C

And we have some blogs so we have. uh This is where Network observability comes in, so we do have Network observability support in 412.. It is also within the UI, and um it is, there are a lot of really exciting things coming up for it. So that's uh if you're interested at all in this I can share these blogs. These are all on cloud.redhat.com, uh and so we have a lot of features there.

C

uh We have more details on logging, five, five and logging, five, six uh monitoring for 412., so lots of things that we have been working on um and lots of these blogs are. These are all public blogs. So you know everyone's welcome to read and ask questions.

A

Have you gotten any feedback from customers about, like the new logging stack compared to the oh, like the you know, I had have you like reached out to customers and done like surveys or whatever and.

C

A

Hey so how's your efk experience versus now you're, looking Vector experience are they like, like.

B

A

That's it's me or are they like holy crap? This is the greatest thing ever we.

C

Have um so once again, the answer is the new says positive for a couple of customers. They are, or users they're, just so deeply embedded into a efk that that is what you know, that that's that's their bread and butter and that's that's fine.

C

um What we're hearing from other customers is that fluent is not is they're having trouble scaling it, um and so the the move to Vector has been beneficial for them.

C

um We're having a lot of people who enjoy the vendor neutrality for Vector, so getting getting logs to different places. um So yeah. Overall, it's been a pretty positive experience.

A

So I guess ultimately like the big challenge going from like the the traditional logging stack to the new logging stack is going to be that that migration from uh efk over to I don't even know if it's a migration. It sounds like it's more like a uh like replace, um but going going into looking right, moving away from efk exclusively and going to Lucky. Is that yeah something about right? Yeah.

C

It's a very uh they're just so different, um so that is a it's really a a clean break if you're going to if you're going to move to from one to the other, and that has been uh the probably the trickiest part, is breaking from one and going to the other um and again many customers are embedded in the efk, but um for those who are making that that switch and going to Loki and Vector um the response has been positive.

C

So it's been a good, a good change.

A

And we got a question earlier and I just want to um it. Was he looked it up and it looks like like Phipps has not supported. Do you know and I'm not sure if you touched on another I'm sorry, but you know like when Phipps might be supported for Loki in Vector, so.

C

That is something that we are examining um within openshift. So that's a I. Don't have an answer to that. Yet that's something that we are looking at, um because there are open shift fifths pieces and there are logging fips pieces. So it's we have to. um We have to scope them both.

A

C

Is so it's a it's a complicated answer, but.

A

We're looking always those right.

C

Yeah. Sorry, yes, but.

A

It depends yep Maybe.

B

Going back to that previous statement of uh you know, the modern technical answers are almost always uh you know yes, but or with these.

C

B

C

B

um Let's see uh Andrew, um can you break down Loki's sizing for open shifts? How does the 1X size configuration relate to the number of PODS spun up instance? Failure domains, Etc um I, will admit, I know nothing about Loki sizing, so.

C

That's something that we can probably um take offline.

B

Okay, so um Andrew, if you'd like uh follow up with me or Johnny, just andrew.sullivan, redhat.com or Johnny j, o n y at Red, hat.com and we'll make sure to Loop Jamie in and and the rest.

A

B

To help with specifics on on sizing.

A

Yeah Andrew's a red Hunter, so.

B

Yeah, um yes, I am but I know that you were talking about Andrew, bicker and uh so for the uh person I'm, not even gonna, attempt to pronounce that username.

B

um We we will post the slides up to speaker deck, which is where we normally post size and stuff like that. So I posted the link there on Twitch. Hopefully it got spread around.

B

um So Khalid reminded us about the network observability operator.

C

B

C

Network observability, we do have that in openshift 412 we have Network observability included within the uh the UI now and so what's coming. There is a topology view where you can view the topology of your network within openshift, um and it's it's really I need to dive into it more deeply. I don't have a lot of experience talking about it, but uh this Blog has a lot of good information about it.

C

So once that's posted on speaker deck, there's that- and um you know if one of the things I wanted to uh touch on before we ended- is if there are any questions about the observability platform, so logging network observability uh monitoring, uh UI any of that uh our email is observability, Dash, PM, red hat.com.

C

So you know we're happy to take any questions and send you know documentation answers that we have. If you have requests for us, something that we haven't thought of yet you know we'd love to hear it. So we're always open at that email.

A

Famous Last Words.

C

Everyone else on the PM team now is just like no.

B

So so Jamie is the network of observability stuff. Is that dependent on ebpf, which again ebpf is a technology that I have some awareness of, but not a lot of knowledge around so I I may be speaking out of turn here for.

C

The network observability- yes, yes, okay,.

B

Yeah I need to uh catch up with Mark and Mark Curry and spend some time with him and and learn more and becoming smart on that yeah.

C

The more uh the more he presents and in forums where I'm also listening, though it's a really really interesting topic, so they're doing great work over there.

B

Yeah I know Johnny and I have talked about wanting to bring them on and have an in-depth conversation here on the live stream, because I think it's something that a lot of people are really interested in, um but I also think that ebpf is one of those topics that it's complex and at least to me it's complex. So anytime we can help bring Clarity anytime. We can help get some additional details out there, that'll be good yeah.

C

It would be um he's and he he presents it so well, so.

B

Yeah yeah uh yogesh is anyone upgraded from 4.8 to 4.9 um I get Ingress, but when I uh yeah so for Ingress, specifically openshifted does not use Ingress. When you create an Ingress, it converts it over to a route automatically um so that that's probably why it looks different than You're Expecting, but I would suspect. Is that and I'm? This is off the top of my head without checking or verifying the Ingress API that is being removed is probably one of the earlier ones like a V1 alpha or a V1 beta, or something like that.

B

So just check the double check, the version that's being removed just to make sure circling around all of that being said, if it's a service, that's provided like one of the core openshift services like we, we fix that automatically. You shouldn't have to do anything. It's only for things like um if you're, uh if your application is deploying something or your, your users are deploying something that uses one of those removed apis.

B

That's where you need to be concerned, foreign yeah I like networking stuff too, but it gets to a point where it it goes way. Above my head, uh I was having this conversation with uh Peter ladderbach yesterday, because we were talking about storage and all the stuff, that's going on with storage and how it works with virtualization and how this changes and- and he he made this tongue-in-cheek comment about you know- maybe I should have been a networking. Guy storage is complicated, I'm like well.

B

Networking is uh there's a lot of stuff going on in networking, to say the least, like you know, and it's even when I think I start to grasp Basics. You start to introduce things that happen at the clients like ubpf right, which is a kernel level, module type of thing, and you know, let alone getting into all the different routing protocols and like yeah networking I can talk about vlans with some degree of confidence, um but yeah it'll blow your mind. Yeah, there's a reason.

B

Why, like you know, you very rarely encounter a network admin who wears multiple hats right, usually they're, just they're the network admin, whereas like VM admins, is like oh yeah, I do storage too or I I'm. Also, the windows admin or you know um you know Network folks- are usually just Network folks I feel like that says a lot about. uh What's going on there.

A

Yeah I think traditional networking is very much like it's. It's conceptually easy to grasp right like especially if you're technical, like I, think it's it's not bad. It's like when you start getting into like the sdn and then the different layers and stuff like that. That's on my oh, my God. What's happening right now, I'm sorry, Jamie, I, cut you off! I didn't mean to do that. My bad! No.

C

No I was just gonna, say: It's Magic. This is Magic.

B

It goes into a black box and it comes out the other side. um That's right, foreign So, for anybody who is watching us. um If you have any questions, you know, please don't hesitate to uh to submit those. uh You know again we'll do our best to answer questions and stuff like that um I'm trying to think of of other things that we might want to talk about, or we might want to ask about from an observability perspective. I think like any time there is change.

B

There is some um and I don't mean this in a bad way. There's some trepidation about it. You know we as administrators, we we are.

B

um We have learned to fear change um and rightfully so in many cases and I think logging, you know in the change from elastic and fluent to Loki and Vector is one of those where, like logging, a lot of times, we have legal requirements around what's happening there, so it's understandable that there's some trepidation, so I I. Think from my perspective, one of the questions I would have for you, Jamie is one is have we officially deprecated the elastic and fluent solution, or we.

C

Have not yet so it is in the it's no longer being enhanced, so we're still doing bug fixes, but we are no longer enhancing with any features so, but it is not yet deprecated. Okay,.

B

C

Of the things I'm sorry to interrupt.

B

C

Of the things that we're trying to get the word out because we don't want people caught by surprise, logging is a big deal and, just like you said, there are a lot of legal requirements. Often there are a lot of um you know very heavy requirements on customers Partners to have their logging work, so we want to start now. You know we're saying hey: this is coming. You know we're trying to make this as painless as possible, but um you know we are in the process of changing our logging stock.

B

C

So we're very sensitive to that, um and you know it's one of the reasons I'm happy to be here to tell everybody this is coming.

B

So do you have any recommendations around when should customers make the switch um I I feel like and a lot of people ask? Oh, you know, is I I see this all the time of is such and such ready for production? Well, we wouldn't release it as GA. If we didn't feel like it was ready for production, so I I always I, think the question behind that is. Is it capable of meeting my needs right? Is there feature gaps? Is there other things that were there with?

B

In this case you know fluent dnelastic that might not be there with Loki and and Vector. That might cause me to reconsider the migration. Yet.

C

I would say that um and the message that I'm trying to promote is we're encouraging people to start now, because we want to know the gaps before we deprecate. You know it it's we want. We want to help. You know. We know that this is important. So if customers start now making the switch, they have major problems. Then now is a good time to tell us. Before the deprecation is complete. You know we can work towards uh filling the gaps, but we need to know what the gaps are.

C

um Logging is a everyone uses logging, it's a really um pretty important thing within openshift everywhere, and so um we are trying to figure out the ways that customers are using logging so that we can, you know, help them in this migration process.

B

Yeah so maybe a silly question: can you run both Stacks? At the same time, absolutely.

C

B

Okay, I didn't know that I didn't know if it was a A or B, not an A and B. No.

C

You can, and that's actually one of the uh things that we recommend is that you run both and then kind of um blue green you're logging. So you know start transitioning workloads over to the other, so that you can understand how it works um and make a smoother transition. That way.

B

um Rick Rick's question here: um Johnny IC responded to that I didn't know. If you had anything to add there about PIRA.

A

No I was going to ask Jamie the question um so um are: is there any integration or uh yes, how are things going with SLO Focus monitoring, so service level, objective, uh monitor, select from an F3 perspective, um so reducing everything else, but also bringing solutions that like uh hero into the picture, I'm, not sure what PIRA is I, don't know if you do Andrew or Jamie I'm.

C

Not sure what that is.

B

No yeah I'd have to I'll look it up and and learn what I can in five seconds but um as any alternative to OCS like we had earlier uh cluster, because infer architect doesn't want to use OCS, which is utilizing three ec2 instances, um yeah um so yeah. So first, just a minor terminology correction, so OCS was pre-open shift.

B

Four six I think four, six or four seven they made the change to odf openshift data Foundation, regardless openshift, four or or OCS odf in all of the four versions is ceph based instead of cluster gluster based. So it does have you know it relies on the replication factor of three. So for every byte you store, there are three or three for every bite you store in odf. There are three bytes stored on the various disks that it uses so yeah I get.

B

um You know, maybe some frustration for the people paying the bill um on why that might be considered call uh expensive, so Alternatives.

B

um So of course we have a bunch of Partners who offer storage that can be deployed directly into openshift um thinking uh peer report, Works among others, um gosh. My mind has gone completely blank, um but there's there's a bunch of them that come to mind, in addition to there's other external things that you can use. So, for example, you can use EBS directly inside of AWS. You can use things like netapp's, FSX and I. Think is the name of it.

B

um You know all kinds of different things that you can connect directly in there. It doesn't have to be odf, OCS storage inside of there um Colleen. You need a huge resource to do that. Yeah, yeah you'll need double the storage space and double the you know all of those things. So yes, um but I think and Jamie again at risk of speaking for you I think the point is you want to verify you want to validate that it is going to meet your requirements.

B

It is going to meet you without necessarily risk of missing something. If you will correct.

C

B

You're, okay, with that risk or you you know, you're comfortable, that the new solution, Loki Vector, can meet all of your needs. Then maybe you don't need to do that. You know entirely up to you.

A

And and Rick clarified about the uh the pero thing and it sounds like it's a uh aggregated view of uh Services they're doing in terms of SLO, so you mentioned that grafana may have like a similar service. So like is there anything within the logging stack that uses the uh like an SLO type, metric or or feedback e.

C

Let me possibly not within the logging stack itself within the UI, possibly, but uh let me take that back and do some investigation I'm, not sure if it's off my head.

A

Yeah, hey Rick! If you want to shoot me, an email shoot me, your uh Andrew, an email or post an email and we'll get it over to Jamie.

B

Yeah yeah uh does reading both increase the load on the cluster. um Yes, uh so I I I would assume that you know the the collection and forwarding agent will consume some CPU, which will scale according to how much log data is being generated.

B

um Additionally, you know, Loki is deployed into the cluster, so it doing its thing will consume resources, but Loki is in infrastructure workload right so yeah you can put it on info infra nodes, so you don't have to you. Don't have to use entitlements for that, but you would have to account for that in whatever your underlying infrastructure is you know, AWS or vsphere, or whatever additional resources used?

B

B

Yeah slos yeah I. That would be interesting to know whether I I I'm, putting all of these things together as you're talking through it rick of you know, being able to Define slos and then utilize. The data that's being collected by Loki in order to validate those slos and alerts based on whether or not they're being violated.

B

C

B

An interesting one I'd be curious about that as well. So we'll dig.

C

That up I'm interested investigating that.

B

um What's the best way to stay up to date with everything Red Hat related um I, honestly I, don't know because uh I'm I feel like I'm in the thick of it and I struggle to keep up like uh I. Think it seems like just about every day is somewhere inside of just just openshift the openshift world.

B

There is a constant, a constant stream of emails, so if this got released or this has been updated or this has been fixed- or this is going on- um you know let alone, if you're talking the entire Red Hat portfolio, uh so I'll say a couple of different things. um So one if you want like low level the things that you're using like you know, this patch has been released or these bugs have been fixed.

B

um The customer portal access.redhat.com and specifically the Errata, is a great way to do that, but that's very much a fire hose of low level information at the other end of that spectrum is going to be like the press releases on red hat.com, where they're only going to hit the highlights. um So, for example, I, don't I, don't even know if we do a press release like a real press release for every open shift release um it's only when there's like significant changes or significant impacts to different capabilities.

B

So somewhere in the middle. There is probably where you'll find that, and it's going to come from a bunch of different places. Cloud.Red hat.com blog is a great place to see. What's going on, the PM team does a great job of with every release. There's always blog posts and I don't mean every release of like openshift, but every release of every component um Jamie's laughing, because I think you just did one of these blog posts like every time. There's something that changes. There's a Blog.

C

I'm gesturing towards my screen, yeah.

B

So that's a that's a great place to look and to keep up with things as well as there's a number of folks who post you know technical things on there as well. You know hey here's, how to do this or here's what this means type of stuff, um so I would say that might be the best way to keep up with um off the top of my head um kind of that middle middle ground between high level, press releases and low level bug fixing Errata stuff.

A

Yeah, the other one too, is I would say: look it keep an eye on developers at redhead.com like Burr Sutter does a good job with like the definition stuff of like getting getting new things out there, getting people looking at new things and uh or like hey, there's this new capability that we kind of like played around with, and now we have a demo on it. So just keep an eye on that too. Yeah.

B

I think that's another one of those examples of um which one of the the laws you know like there's Moore's Law, Maslow's law, all those things there's a law. That's like you know, the the structure of a product will reflect the internal organization um so like openshifts, like the things that come out of openshift all go into cloud.redhead.com Blog, the stuff that comes out of the cloud services, business unit and sort of the the uh Red Hat run times goes on to developer.redhat.com. The stuff that comes out of Rel goes into redhead.com.

B

So if you're interested like portfolio wide, there's different places to go yeah so.

A

B

Yeah, um let's see there was a question from Khalid I think.

A

Johnny it was uh which produces more storage, is it elastic or Loki or which, which uses more Source I'm. Sorry, which uses more storage, elastic or uh Loki.

C

Not certain I think that.

C

That it's hard to have to my head.

A

B

Off the top of my head, I, don't remember if elastic uses the same sort of protection through replication scheme by default, yeah.

C

I, don't recall, that's why I'm having a struggle answering the question.

B

So um well, I know we're at the top of the hour um Jamie I forgot to ask you before we started. If you had a hard stop on anything, I do have a hard stop today at 12 15, so 12 15, Eastern.

C

No I don't have a hard stuff.

B

Okay, okay, good I was hoping we weren't keeping you from going on to something something next, um so with a few remaining minutes um for any questions that are out there, please go ahead and submit those over we'll get as many as we can in before we have to drop from the live stream. um But that being said um really, thank you so much Jamie for joining us on the stream. This has been super helpful and very educational. For me, absolutely.

C

Thank you for having me I, really appreciate the chance to get the word out.

B

Yeah and uh I think I said before that, we'll post the slides that have the links and everything we'll post those over on speaker deck, um so I posted the link here in chat, but it's simply speaker.com red hat live streaming. um So all one word: no spaces. Red Hat live streaming.

B

um It's also where we post the slides for like the what's new and what's next sessions and all that other stuff, so yeah we'll make sure to get all those links out to everyone. So that way you can access those and learn learn more yeah. This has been awesome. This has been very educational and I'm sitting here thinking, um so one I miss all of the artwork that I normally have behind me, but I'm admiring the stuff that you have behind you and it looks like that's a like a Wookie or something playing.

C

Yeah so I play bass that is Chewbacca playing a bass so.

B

Yeah, nice yeah, okay- oh that's great! So.

A

B

Eventually, I'll be back in my old office and I'll have artwork again but I miss having something in behind me. That's not you know. Just the the scrolling red hat um and Johnny just lives in a hallway. Apparently so I do I. Do yeah yeah hey for you found a place to that's kind of consistent for a while there you were changing like every week, you'd be in a different room in a different place or the house, so dude.

A

80D is real man like I, can't sit still yeah I'm all over the place, so.

B

A

B

I I feel your pain uh all too. Well, that's why I have to have a a specific setting like for me. It has to be consistent every day. It helps me um yeah, so anyways. um Thank you everyone. We really appreciate you joining us today. The the chat has been phenomenal. We love getting questions from you all. We love all the interactivity.

B

um So thank you so much for that uh again. If you're not listening to us live or if you're not watching us live, uh you can reach out to us anytime. uh You can reach me at andrew.sullivan dot, red hat.com as soon as I find the button to click here, to show it to you or again, practical Andrew across all the various social platforms. I will admit that sometimes I lag behind on things. Email is a real issue these days, especially after two weeks away, I'm still way.

B

Far behind on my email, um like probably 60, plus conversations behind and each conversation is multiple messages. So my apologies for anybody who has sent me something and hasn't heard back um and uh you can also reach out to Johnny um Johnny j, o n n y ireda.com, so yeah don't hesitate to reach out at any time um so yeah I don't see any other questions that are popping up in chat.

A

Let's do more streams on networking and uh so I just reply back like yep. Definitely on the on the roadmap to do at least a few right like it's a big topic so uh we'll definitely get to it.

B

Yeah, that's a good point. um It's been a while, since we did I want to say, uh like 60 plus episodes ago was the last time we did any kind of Deep dive on like sdn and stuff.

A

B

We did talk about uh NM, State uh and doing kind of low level. You know host level Network config, but um so that that might be interesting to you. You guess if you want to look up the um uh the playlist and look for that episode.

C

B

I think it's definitely time for us to go back and revisit things, especially now that openshifts or ovn kubernetes is the default sdn as a 4.12. So it seems like it's a good time to go back and and really dig into what's happening there, um so yeah. So thank you so much for joining us Jamie. We really really do appreciate it.

C

Yeah, thank you again for having me. It's been great.

B

Yeah and uh with that I will uh give you last words: Johnny.

A

Yep uh Jamie seriously, like you, you said that the network observability team were doing good work. You guys are doing good work too. You know, so it didn't want to didn't. Want you to downplay like your the stuff that you.

C

Guys were playing no, no, no, no. The logging team is a great word. Yeah.

A

I'm excited I know like my old customers. You know I'm sure that they're excited about these capabilities that are coming out. So thank you very much for coming on and talking about them.

C

Yeah, thank you.

A

Andrew, let's continue again yeah.

B

Yeah all right, bye, y'all, see you next week. Thank.

C