GitLab #EveryoneCanContribute cafe, 13 Sep 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: 54. #EveryoneCanContribute cafe: Pixie for Kubernetes Observability

Description

Blog: https://everyonecancontribute.com/post/2022-09-13-cafe-54-pixie-for-kubernetes-observability/
Pixie website: px.dev
Installation: https://docs.pixielabs.ai/installing-pixie/install-guides/
Scripts: https://docs.pixielabs.ai/tutorials/pxl-scripts/write-pxl-scripts/custom-pxl-scripts-1/
Dashboards: https://docs.pixielabs.ai/tutorials/integrations/grafana/#create-a-dashboard-panel-of-pixie-data-select-a-pre-made-script

Mentioned articles:
- https://medium.com/@isalapiyarisi/getting-started-on-kubernetes-observability-with-ebpf-88139eb13fb2
- https://www.containiq.com/post/decrypting-ssl-at-scale-with-ebpf
- https://aws-otel.github.io/

Events:

- Kubernetes Community Days Munich: https://community.cncf.io/events/details/cncf-kcd-munich-presents-kubernetes-community-days-munich-2022-1/

A

Okay, now we are back. I lost progress on how many meetups we did in the past, but.

B

I think we are on 55 right now. Oh.

A

So it's like in the english version.

B

So we have also a german version, so I guess we are about 80 or something like this.

A

Yeah, probably we did a little summer break because nobody was available and um figured that it might be a little too much. um But for today we we thought about like observability and security themes or topics, especially since uh nicholas influenced uh ebpf and pick the project which we will talk about in a minute um at container days um and yeah. I'm looking forward to learn something new today.

B

Yeah for myself we can start so probably.

A

I think you have prepared something at least and.

B

It's a little bit pretty much. It will be like um more like an adventure, so I have only installed with cli nothing more and prepare the cluster because we later need the cluster. um So let me share my screen and I need to check if I don't, if anything, probably.

A

It makes sense to start with the documentation of pixie, which we are talking about now,.

B

Give me a second: I will check for the right side object. Your screen, probably some whole desktop. Now.

A

B

Be seeing android is it. I.

A

Don't know just it's for.

B

A

Yeah, it's a really short domain.

B

Yeah but funny is there's also cloud service offering, and this is called wordpresspc.ai. So it's also going to the same project. We will talk in terms of what you can host on-premise and what you can use from the project later. um Okay, I need to move this.

A

Probably for context pixie was, I think, a company or a standalone open source project and was acquired by new relic two years ago, or something like that um and has been um contributed to the cncf which was shown at the bottom. So it's a sandbox project now in the cloud native computing foundation.

B

Okay, then you're more informed than me, so I know that it was built up from some big use case that they used and the most interesting pro where you asking now why we should talk about a new observability tool in the end is mostly how pixy is working in beyond. So since, typically, you would see that you need to instrument your applications for the observability.

B

For example, I've seen tracers see requests and over stuff, or you probably do it like on a different layer in terms of like on a service mesh, um but right now, these days, a lot of people, probably switching to the new shiny thing that stored ebpf, that virtually patriot filters in the end that you can inspect all packets. That find you through your system and these can be used and to write across an ebpf rule.

B

It's like a simple program and you can act on each package that goes through your server and so they're using this technique in the end to analyze your whole application stack instead of instrumenting your application inside your application or getting the observability, they do it on the infrastructure level of kubernetes.

B

um So here we can see directly on the side here how a service could look like. um We cannot see the parts that are currently running. They are also more focused on the sie topics, typically with slos and all stuff, um and you simply like you deploy it, and I want to see if they see slavery, requests things um yeah and you don't need to instrument directly on your um things and also crew is like that. You can also not use it in kubernetes.

B

You can use it on any linux film outside, so you can also probably probably have some services. Typically, I recommend people that starting with kubernetes, not directly learn stateful services in your cluster, because it adds a lot of headache and complexity.

B

More serious would wise run database how you know to run it run it as a service if you use it as service. If you are familiar to run it on a server, then use it as a server, and this is where also pc can help you um for that, in terms of that, giving your observability to the database so that you get the full trace in terms of from the application to the database and so on.

B

um Effing can be in your private or public network um and the most interesting idea that we are using is here the um the ebpf. um So I did not start with all things. um It can be run on every tribe, anita. So for today we will leverage um a short.

B

Eks cluster, so I set it up already and managed hs cluster that we have for playing around a little bit um in terms of with nothing really currently running here. So you can see you have running some services only and it's like a fresh cluster. um I hope everything will work later and with only one note, so right now um yeah, so the main idea how it works is like.

B

I need note to switch my to my other slides on to this side that yeah group, um so that you would have also probably like while we talking about the pixie cloud is like you can have. The um visualization of all multiple clusters can be done, then in the general pixel api that will send all the not all the events, but um um you have like a unified ui. So for that I already also registered. I guess so with it. Let's see.

B

I need to check it later from the zero. uh This is the script that we used for pixelates that'll put emoji for um so this is the login. Let's login.

B

B

um And now it logs in.

B

Takes a minute- um and now this is what you see directly at pc, so we have no cluster currently configured and now we want to um get some visibility and later also setting up the demo app in the end. So for that, let's, but we have a demo okay, it's interesting. I need to check for the getting your first ppc guide, so we have mostly what we did already so we installed with cli, but in terminal you get the cool output.

B

So when I look back into my terminal, you see here that I already run this. I would never recommend people to do that in terms of check please for strip first, that you're injecting and don't do a bash, injecting those strip boards coming from some internet endpoint because probably triple run malicious code on your system. I would not recommend it, but for flexibility, it's like quite good, um and now we need to deploy pc on our cluster.

B

So for that, since pixy comes with an on cdi, so you don't install frame shots, you can directly install it. um We have a ph deploy command. So, let's check um let's check if my cluster is available, so, for example, I press credit notes, yeah everything is running cool um and let's deploy pc and I need to wait. I need to check if I ah I switch on keyboard and now really um checking the cluster um if everything can be deployed.

B

So let's wait what's happening.

C

So basically, you just randomly pick the cli command and it deploys automatically deploys the corresponding kubernetes resources in the cluster.

B

Yeah everything for pixie and probably also so we get lightly guided through. So it asks me if the cluster is correct. Yes, the stress is correct. um I have also um to check so it uses the old m, so this is for operator, lifecycle manager, so it's coming mostly from red head um to insert so it inserts in our namespace.

B

It installs the resizer component in our um deploying the application stick um so for the cluster I already created on the um the cluster with eksctl, so a simple cluster um and prefer a simple mentioned node group of one, so nothing special so that because then it's quite easy to get started with a cluster, so um you need only to say, create cluster, and then you have your customer system um for cloud now. It's waiting um for coming to the cloud connector online.

B

They check um it asks me all also when I installed the cli to connect directly to the um authenticating directly against the pixie cloud. um So that's the reason you need to have appropriate insult, but you can also skip it. um So when I skip this, um you would have probably not visualized component um and let's check what's happening in our ui know these days.

B

B

Okay, we see only pepsi, let's wait, what's happening, I'll wait for pms waiting for pc, pass, save checks, or we see something in here on the ui. um I hope so we'll check announcements. Okay, nothing interesting!

B

ah You see our clusters now appearing, um it still says unhealthy, but let's wait um until it gets healthy. I think it takes a little bit time um yeah and in the end, what is insults um we can look back into the slides that I had prepared over graphic, um so it inserts all the resizer components is mostly for communication between the um different modules and the core component is to get all the information. Is the pitch pixel edge module?

B

So this reads directly out of the linux kernel, the ebpf events or like evaluate for functions of the ebpf, and this gives them the information back to a message to you in the end and then some other service can use it, for example, like for proxy, so from the pc cloud, can then use it in terms of okay? We now know what's going on, so that's how it's currently worked so far, what I know um and yeah. So this will also be data, and you know um I thought so linked it.

B

Let's check also in the meantime, for the tutorials so to get the um it was like when you installed right stream into cloud. Let me check for testing an app um this pc yeah, so um we can also use them later for demo apps to monitor everything um we can check them. Also using the sea light gives us some um available service, so probably some people of you know already um from all days the stock shop. So this is like.

B

B

It's like a microservice demo.

B

um So it's like a demo application that you can see that every service is running so each service or each domain has its own service to see some traffic in the cluster to simulate some things, and probably there's also like another ship from google that you can use um yeah, but there's mostly no difference and the proofing is also that each and then we probably see later the result. White pixel is more strong for instead of like instrumenting our apps directly by ourselves.

B

It's like yeah, probably in our microsoft architecture, we have different programming languages, so some sources are written in java, some written are in node.js, some are written in go probably now we need to instrument all the applications, but with pixy we can abstract it a little bit put it back onto the infrastructure, layer and pixie will use and the job. So no, no, although developer needs to do the instrumentation now on java developer, needs to do the implementation. We put it back to the infrastructure there.

B

um Let's check in the meantime how fast it goes waiting for pc to pass the health check. Okay, I hope it doesn't. So I'm not restricted in my environment in terms of the.

B

ah So we see already the first error message.

B

A

B

Yeah: okay: let's check.

B

In pending mode and.

A

A

B

Resource should be not a problem.

B

uh Let me switch to the player name, space.

B

That's interesting that they're still pending are they probably interested in waiting on the metadata servers and the meta set. Let's just check the second.

A

The system volume claim not found.

B

um Okay, so since we don't have any stocks that will be interesting.

B

But why wait try to describe.

B

Let's see, let's check.

A

Did the aws cluster not provision that.

B

By default, yes, but I think they changed it, okay, so now the error message is here mostly that they say waiting for volume. I can create it by an external provision about the ebs, csi or manual by system administrator. Now I need to think how we can.

A

I would paste the message into google.

B

Yeah, well, that's boring. We need to think before we copy it.

C

Yeah, no, you need to add a px in the end, otherwise you could get results for non-pc solutions.

A

Right, I'm not sure if that's a pixie problem, I think that's the problem.

B

Is the storage of our cluster, but I'm.

A

B

Cross lets me check something.

B

I can ever not install initial volumes.

A

You are using the web interface.

D

B

Yeah yeah, but I can inject in terms of eks, supports us add-ons, so we can um install here the ebs csr driver that was mentioned here um and just like the um controller.

D

B

Yeah because in ah I know it, uh there was a breaking change into an ets um on 1 23 and we deployed 120 feet and you need to have the ebs csi driver.

B

Otherwise, storage is not working anymore because in storage providers on eks, since mossy hunter 1.22, the ebs provider of the volume provider was built in the kubernetes control plan. So that's recording entry projects and this drop moved out into 120 feet now for persistent volumes. You need to run the um for dynamic provision like what we see. That's the reason why it says external provisioner um that we need to have it here.

B

um So then that's. I hope we can easily pitch it. Let's edit and hopefully it will work by itself um and we need now to have the storage driver, so the ebs search driver is like an independent driver, um but let me check because ctl wheels.

D

B

That I read the documentation because we had the same problems and production clusters when we.

B

But I need to have the rovers.

B

B

So, typically, now all the project coming up or we you should I here you see the slide, the controller that we are currently running, and this is now for attaching the volumes directly to the node. Now we could have like global. In terms of that, we.

B

Need to check for the alerts.

B

When you're still just destroying the name yeah, we should probably wait a little bit and it should probably resolve by itself, since it's self-healing.

A

I think I was wondering if pixie works with k3s.

B

um They have a documentation for minitube.

A

Because, like then, you could be running it in civil cloud, for example,.

B

ah Let me check that insert.

A

But maybe it's not officially.

B

Therefore, they're using chase zeros, what's that, it's like um k, k3 is still smaller. It's coming from the mirantis people.

B

But I would say it's a little bit the same, ah so you here you see, you provide also like a search class, so we need to fix it. So.

A

And probably this is in the eks guide as well.

D

B

Probably it comes out, but I think never no one deployed it with 123, so that could be more of a problem um and on mini cube it says blah blah blah.

B

I mean it probably comes out so with 54, it's storage provider. So it's a simple problem.

A

Yeah this session would be something for corey quinn,.

A

D

B

Okay, now we have further um the controller says that he doesn't have enough permission to create it. um Yeah we're fixing this. Now, it's quite simple or not simple. It's like we, we do not like this would be not best practice. It's not for the demo purpose. It's fine, um but typically what you would do is like. When you have these uh addons installed, um it says you have a service account. Wall is inherit from the node.

B

Typically, you would do like a separate service account to manage those access provisions so to have like simpler, um simpler rules. um We give now the service node role, also the option to create the um volumes, and then it will be fixed so that we don't need to create an independent services. Controller best practice would be create an independent service account role and um yeah. So let me check.

B

Yes, here I am, but can we start enough from here somewhere.

B

B

B

Yeah, we need to have like these policies um adapted to our roles, so that we can create the nodes of csi things, um so we're copying it and jumping back to the. I am.

B

B

D

Next time I test it.

B

B

Over interesting, which is the right one, not instance, raw, not cool, so and now we add um an inline policy. As I said, this is not best practice. We fix it now like we should work around.

D

B

C

ah You have a blank line uh line one. I think that's what.

B

Yeah, that's really why I use infrastructure stored for everything, mostly.

B

B

You see, for example, like each node it access to the registry so that he can pull the registry images directly so that you don't need to re-authenticate and also like he has privileges to describe his own instance or all instances. So it's probably also a little bit shut down and also to get access to the ssm is a different topic. But now we give also all nodes the options to create the things by themselves, and hopefully now it will be resolved by the next round. Trip of the volume.

B

But this event is three minutes ago. What is the read time? Yeah? Let's wait a little bit.

B

For ensuring so that I not attached it to the wrong node, we can also look into the ec2.

B

B

um With here or note- and it says, blah blah blah.

B

This is the note, roll um and, I hope, yeah. I attached it to very creative group, so this was a good shot, but then it will be complex when doing those. Okay.

B

ah You see now it uh it shifted from. uh He doesn't find any status he's in pending state um and now it could access creating the volume. And now I guess.

B

Yeah, so the metadata service is now in the running state. The first applications are coming up and now it should work at the end. I hope so, I'm all the traditional guys. If I do something like that, we can just simply wait a minute there. We just you know the um kevin help, stick no all applications all parts for um I would note here what is the metadata, so it's a ticket.

C

Maybe it collects uh like application metadata from deployed services or applications that we would like to monitor or something like that.

B

I want to know if which database it is or if it's not source.

B

Okay, it's now on sales, but okay, it's not like yet a simple stand-up metadata source or like a simple database on okay. How is waiting for our health checks? I hope, let's check.

D

B

Okay, something did not work, but let's check. uh What's the ui says.

B

Face reject credentials, page c of json.

B

Should I do this?

B

I will check with everything.

B

It's really important if it's only timing out.

C

um Is there a way to provide like credentials when running this command, because maybe this is missing? uh It's like the the config for for docker uh registry authentication, or something like that seems very similar.

B

Ambitious, let me check if you can redeploy, it should be now a little faster.

B

So they have something that kicks off.

B

Okay, now it's it should start. Let's check what was where in which file.

C

I think it was uh dot, pixie, uh slash, config, json or something like that. Heading home.

B

Okay, something, sir, so probably my br token expired already.

B

B

Now that's amazing.

C

Oh, the cloud connector had a crash loop back off.

B

Well, that should be fine.

C

Okay, it's gone. I know it turned into arrow.

B

I'll, probably use the configuration power now you see he's calling on the nuts management, and this is.

D

B

Okay, it takes some time. Okay. This is like what you need this mostly we wait and check if you get requests by itself, so since it's self healing it should not- and this is like mostly error prone when you start with it. So when you be too greedy and changing too often, sometimes you need only to wait a little bit until the state is mostly matched.

B

Yeah sometimes some issues: okay, so.

B

B

Let's check what's in the ui, I.

D

B

We see something more, no is it also.

A

What happens when you click on check, chat with us, then.

B

We need to chat with them interesting. Are they using intercom markup.

C

Maybe we can ask them for help and tell them: they're live in the internet right now,.

B

Probably another problem: I need to accept the cookies, otherwise it doesn't work so since website doesn't contract me but you're right, it's coming from new relic. So.

A

Refer to this docs url in the error message.

B

Okay, let's check why we don't get this solved.

A

Too many open files.

B

Yeah but they don't have running in.

A

um Is it lsof to see open file handlers.

C

uh Yes, I think.

A

What is different on mac os and linux.

B

Do you mean that.

A

Yeah but potentially a lsof pipe grab, firename.

A

Yeah, no, we don't want to change that.

B

Can I not see return to your limit unlimited.

A

Yeah but the um for some reason um something tries to read the file too often.

A

B

Okay, then, let's.

B

Check what we can do.

A

Strange error messages.

B

A

Is it no healthy again, no.

B

Yeah, it's okay.

A

A

Yeah, because unlimited for um open files is typically more than the value you have set.

B

Let's check the doors.

A

It seems ah I found a an issue where it says to delete and redeploy the pvc again.

B

A

Read the issue first before you do something.

B

A

B

A

Yeah the zoom chat.

A

Maybe this provide some insights.

B

We don't have the sub traditional.

A

B

A

But it has the same error message.

D

B

B

It's really deployed, I deleted the whole namespace and then check and also no pvcs anymore there. So I checked it now: people's thoughts, all right so yeah, oh wait! I'm there connected absolutely to the right notes.

A

It's redeployed. You deleted the namespace right now.

B

Yeah, I really pronounce so that we get uh so it was easier to instead of installing.

A

B

I deleted the whole name so that the pvc is also wrong, but.

D

B

That the operator will recreate everything.

D

B

But it did not.

A

I'm not sure how, if that's an operate on the background, is.

B

It yeah they have something like with pixie operator.

B

Let's wait: let's wait.

A

ah Yeah there's actually a namespace for the pixie operator.

B

Here page operator, you can see it underneath so and you see now everything will be reprogrammed.

A

Yeah the thing you are not seeing, I was just looking at the install guides in the docks.

B

A

B

The slide like really.

A

Manual, the manual yeah, that's the thing with an ikea furniture instruction manual. You need to read it to do it.

B

No, when you can get it done without the menu you're cooler.

A

Yeah, this works for every other vendor for furniture, because the instruction manual usually is not usable, but for ikea, it's it works. Oftentimes.

A

A

There also is like the sock shop, um which is inside the pixie cli interesting.

B

Hn pc demo, so that what we wanted to try after the health check is passing so now we need to cross our fingers, and hopefully the health check will not pass.

A

It's kind of weird form of debugging.

B

Just for mentality, so that you need to recreate everything. So that's repeatable.

A

Yeah, it kind of reminds me of windows just rebooted, and it's like the german thing is like reboot toot good. So reboot is nice, but.

A

Yeah oftentimes, it's like feeling blocked and it's easier to just delete everything and recreate it from scratch and then go again.

A

We could uh install uh psyllium tetragon to monitor the health state of.

A

And if that is not enough and.

B

Now it's installed, then we have resist operator, debugging, ebpf programs because they are competing against each other yeah. That's what I ever wanted to do so.

C

I think it succeeded right.

B

Yes, it's actually nice nice. So first we look on the ui, because I'm more video idea.

C

Oh well, that was so bright, yeah.

B

Because it was white now we see everything here and.

A

B

uh Yeah, so this might be interesting in terms of you write in terms of pixie, you don't need to write ebpf rules, so you can write, um they call it pxc scripts so and just like a simple abstraction above the eppf rule, and um for example, now we don't have anything like for amp or like, but http data should flow. Let's see, let me.

A

And here you see, for example, like.

B

We never instrumented the application, but we can see how the request is flowing. So we see also relatances.

A

ah And pixie scripts are written using the pixie language, a dsl domain, specific language that follows the api of the popular python data processing library, pandas pandas uses data frames to represent tables of data. That's described in the docs.

A

B

It's like a little bit similar to.

A

Hover because the uh the code looks familiar to.

B

Actually, that's uh yeah, for example like we can also see like http requests. What that's coming out.

A

B

A

This would be cool. You could write your first pixie script. I would probably.

B

Draw first with deploying in demo, applications that we have some data in there.

A

B

So to see like the service and then we can test out what we can use. So what I what I now saw in terms of okay data retention- I don't know, but I don't know where it is so ah so interesting here is like I here. This is what I now see as really two of things.

B

Okay, we don't have any escrow queries, but we had something dns.

D

B

D

A

B

For example, you can see also each source which uh dns corey was made from rich servers, and they see also the redress header of the screen. Where he's trying for so- and you could also do this later than with http requests.

B

So we saw it already so here we can specify some like posts and to check what's happening there, or we can also do like.

B

I don't what this is, and here we, for example, see normal http requests what coming from flow to flow. So simple data requests.

C

So the basic idea, if I understood it right, would be to now yeah we're we're able to deploy any application and it should uh yeah like be displayed in within the pixel ui. uh Without doing uh anything more so we can create own pixie scripts. uh For I don't know aggregating.

D

C

Metrics or uh what is the relation between the script.

B

Okay, so what's my assumption now.

C

uh So basically, so uh my only experience I'm, I only have fairly limited experiences in the observability and monitoring context, but I uh I recently had contact with prometheus and grafana um and yeah. I tried to set up this one uh and there I I went for like the service, monitor architecture, where you add service monitors for your deployments and the problem operator will fetch the metrics via service monitors by a service monitor layer. um So this would be the same thing without having a uh like a service monitor for a resource right.

B

So that's right, that's a quite different because you are talking about metrics and we are talking about more about traces of data and lots or events so because a metric.

D

B

The end slide only like, for example, a metric is like okay, I get 200 on the following endpoint, but just like not, you cannot see which data is in there. So you get only like okay.

C

So it's like an indicator.

B

It's only an indicator: it's not like um the real data, so here we can really see the process data that was happened.

B

um ah Okay, so, for example, like I don't know, okay, ip doesn't work, but jvm will be later interesting and, um for example, we can also see like mysterio data, that flows through the cluster and all the stuff, um so with metrics. So there is like my favorite.

B

D

B

So I really can recommend so where we are fitting um fixing. So mostly observability is mostly of a fear of kinds of things, so we have one pillar of metrics, so these are mostly like really lightweight so there and they are gradable. So that means that you do um doing calculations on top of it to get light. Notifications of the state of your system. Traces are really the requests that are happening, so this can be like on http base.

B

This can be on tcp based software supplied with sql queries and all the stuff, so they are more, they are more heavy, have more contacts and then, in the other thing, whatever your nose is probably locks. What we also can call is equal to an event in the end, so that something is happening and you get a lot of more context. So sometimes people are trying to um pretend to doing like logging on the transfer words and using this also as an event system.

B

But most people know these days, starting with tracing, um and today they have the event data in this. So to see what how is communicating from surface a to service b and so on, I should need to be midnight so bright light. I think it's triggered, but it's what I recommend is now yeah.

D

B

Here you can also see um in the arctic, so also, what is the volume in terms of how many data needs to be stored? So metrics are quite simple and very compact, so they can be sold easy. You don't need to have a heavy infrastructure for storing locks. For example, you know, probably you want to store all your lots of your application. You need probably to spin up an elastic surge or something like that. These are really heavy.

B

So these mostly these systems, are mostly for most load systems in your environment and um the other thing is like trace a little bit more compound, so they are a little bit, not so heavy like like normal, lock events, but they're, also more heavier than metrics they're in the middle of them. So and so, for example like when you want not to make an um correlation between this. So, for example, you jump from your metric into a trace or the trace can show you um the metric to which it connects to.

B

So this is what we're doing when you put it on our metrics. It's called examples mostly that you put more information on that, so that you can typically do a drill down. So what would happening is like um that. You probably get an alert from your metric, because the metric is mostly freaking out um because the threshold was missed and then you have information.

B

Okay, please look now and and these in these traces that I currently could issue it, and then you have the correlation back to it, and so then you jump in from top to bottom.

A

Yeah, and so it becomes becomes even more complicated when you're looking at more data, which can be seen as like. Observability data like um error, tracking or profiling, has been like suggested to being added as an event type or data type to open, telemetry and even more things, basically to look at and coordinate.

B

The good thing that I really like about physiologists every time for which cluster should deploy to, because sometimes when you are connected to multiple clusters, you probably deploy it to the.

A

And another thought around, like xm plus I've, seen that there was a pull request merged and also reverted later for open telemetry, so that when promisius is sending the exemplar data so like the span id and the trace id uh in the metrics metadata, um it should get automatically like merged with the traces or the trace data which comes into.

A

I think there is like work underway to to finally make this happen and also an open telemetry.

B

And why you want to do this and to have like when you're looking on traces is mostly you want to have you want to know, what's happening from the beginning, so probably in your ingress, you um generate the first request id for the redress, that's currently happening and then goes through the whole system, the same id and I can easily track what's really happening until to the data. So what's probably phase so this mostly, but also people sometimes missing in terms of like um creating it. I hope no, it will still all the stuff.

B

That would be no interesting.

A

Let me quickly share my screen: um yeah dude.

A

So I was talking about a little bit about traces at container days last week, and one of the things I tried was to use the instrumentation within open, telemetry and nginx, which kind of is loading an nginx module similar. You can also use it with apache there's also a web server sdk module, and the idea is that you can like analyze the traces and also figure out, compare traces figure out differences, and the good thing is also is you can natively deploy it into kubernetes as well, uh using the there were their operators?

A

For um you see, this is the project I kept hacking in for telemetry for jaeger, and you potentially also need the serp manager, something yeah, so this can like being deployed um and from the the nginx module. I've created a deployment as well, so this is kind of the everyone can contribute, cafe, contribute service that gets deployed, and the idea basically was to not only have like instrumentation now for the traces um but to generate uh or use chaos engineering to generate slow requests.

A

So at a certain point, you probably want to alert on traces which either take too long or are missing some context or something else, and I think there's also potential for for the collectors or for the tooling to like provide more insight to provide more correlation. To that.

A

The thing I tried was like a stress test, then providing the the details that this request takes shorter than the other one and another thing- and this is the last slide I'm showing now um there is an idea in uh to create these aggregate aggregated uh trace metrics, so that when traces are being sent to open, telemetry and jaeger that you can again generate metrics out of that and have even like more valuable insights.

A

What's going on, so I would say the future is bright. We need to learn many things um and yeah. It's moving fast.

B

Okay, in the meantime I needed to scale to beneath this cluster manually. We don't have notice data here or autostream is not working as accepted.

B

um This is another topic, um so now you probably can see now that we have like a lot of more services in the structure and, for example, the payment is starting. So um for that you can easily check it.

B

And you see that it says: hey your notes, don't have any cpu anymore. I hope they don't use so much um and they scaled out a second node. So we have now um two nodes so they're currently running. Now all the containers will be created, so we have a full demo up running.

B

um So I hope this is enough, then, for the stock shop cool, let's wait a little bit.

B

So you see also, then we have no second node, so the second server instance and now look back into our cluster.

B

So we should now see a lot of more things happening.

B

And right not ah here, you see also that for services now a little bit more torching.

B

Okay, these are the old.

B

Let's use the sound, is it the same? Okay, it's the same that we've used before okay, so now we can see uh http, and you see now a lot of modern mutations now happening between all the services.

B

B

Do we have also knowledge emulator, ah nice and, as you see so, for example, like um that's, not interesting, we have also the sweatshops comes also with um java applications and java is like a virtual machine, and the virtual machine um is like also instrumented, and we can now see, for example, how the application was started with which configuration and we never instrumented the java application. In the end. I hope no also that we can see now more interesting things like.

B

What is this most http data? Okay? There we see the front end.

B

um This is not so interesting, I'm looking more for the database data ah cool ah here, and this is not interesting. We never instrumented any application to do that, but um we can now see also, for example like when the catalog drop is storing the database and um we can see which command was used.

A

um But this only works if the tcp connection to my sql is not encrypted right of.

B

Course yeah, so all you need to decrypt it on the fly. I don't know if it yeah, but this is the same on ebpm ebpf is the same right here. You need to decrypt it. um You need to decrypt it. This is the same. That would apply if you use celium and you want to block http servers. I want to see which traffic is in the https package. That will be technically the same, um and I guess we have now also a lot of more stats.

A

And the reason I asked and I've also shared the link in the chat.

A

I've recently read an article how to decrypt traffic with ebpf in kubernetes, which which is super interesting, but also rank a lot of security alarm bets.

B

Yeah, that's the trade-off that you need to say when you want to have so data, and you want to encrypt the data and transit. It's mostly that you don't trust others.

B

But if you are a trust owner, then I would see it not as a problem since you are provided with data, but it's mostly that an attacker could come from outside when you are not in trusted environment. So cloud is not a trusted environment, so you should encrypt every database that you're using and not also in transit, also address so that all the data that is stored in the database is also encrypted on a disk drive. So that's it.

A

Yeah, the article is a little longer to read, but I found it very insightful. Okay.

B

That's my homework for for a night, whatever it is um yeah, but now we have all the the um applications running. That's quite cool, let's probably try um to check. What's more inside the pc.

B

Oh, I feel it disabled.

A

I'm wondering if there is a pixie script for the sox demo.

B

Yeah they have flight. Let's check your continuous monitoring.

B

You probably can use it directly monitoring the demo application.

B

Page namespace, okay and then click on which simplest, okay,.

D

B

Check on the following script repeats: namespace. Please.

A

But they do have a tutorial for database query profiling.

A

ah It's in pxe101 tutorials in the documentation.

B

Okay, namespaces, we can see only the data for this brief transcendent.

B

ah That's what I looked for, it's not centros.

B

So now we have, we have also like when you click on pc namespace. You can select in a separate namespace. We now use our demo application here.

A

If you open the tutorial, I showed you I'll link to you, which one this one.

A

This one, um which.

B

ah Maestro sets yeah. Okay, would you enjoy that afterwards? Okay, let's do the first one that we saw so I clicked on namespaces, I clicked, then I selected under the namespace or pixiestockshop, and now we get a total services and we can see also the latency profiles of each server so fast99 for when means like 99, 99 percent of all requests states, one. Second, we see the um http request throughput and also the error error rate for each service.

B

I hope um and also we can see which application uses like which big size and probably now, when we click on the uh front end.

B

It shows us the latency the response data and now we can also see um each inbound, hp, traffic and what's happening, increases the error rate, um and we can also see the data.

B

In terms of the front and probably did a request to slash card- and it says yeah, it doesn't work, you get a 500 bit and you can see also which data was uh asked for in a post. So you can see the full request body um regarding the maestro everything let's go to maestro.

A

There's a drop down uh from the script io. The script shows latency error throughput over time for all my sql requests in the cluster.

A

Select the drop down arrow arrow next to the port argument.

A

B

I did it master so when I put my square sets in there okay, I know I broke it.

A

Next type px-sock shop, slash, catalog press enter to rerun the script.

B

Okay, I think when it doesn't find anything, it will show you all. So we see here already the ph job, so I try to copy it.

A

I'm not sure if copy paste works comment.

A

Yeah, but only if the ui allows it.

B

Let's check my stress: that's.

B

I measure some the words. No, I don't know why, but it works.

A

Yeah because you selected the pod.

C

I think that's exactly what missy wanted to do before.

A

Did I do that, I'm just reading what it says in the documentation.

B

So we can see here now.

A

Hi, okay, it's limiting on the port.

A

uh Interesting, probably it should be auto completed because the pod name is different to yeah the catalog.

B

D

B

Here, for example, now you can see that vegetalog makes a request to the catalog db, and then you can see also the asteroid tree.

B

For the full application, so what the application is doing.

A

And then there is something about normalized queries.

A

Oh, this is complicated.

B

Okay, this is like another exercise, so when you are having it running, you know what you do, then it's more simpler, too.

A

Yeah, it's like when you're, for example, using um parameter bindings. We have to the question mark placeholder and you're kind of reusing the sql statement, all the time, which is which often is much more um performant, because the server caches or the server could potentially cache the the query execution planner strategy.

A

Rather than passing and evaluating the sql string statement. All the time.

A

But opinions differ on that strategy.

A

Individual full body request. Okay, this is what we have been looking at and oh, there are community scripts available for postgresql radius and cassandra.

A

Pixie has a feature which is called always on profiling,.

A

You can generate a flame graph.

B

I mean normalized trees. It's like a white trend.

A

ah Okay and then it's something else.

A

It's been quite some time since I was required to design a database scheme which I'm fortunate, I think.

B

Which is, we already saw.

B

As they mentioned, smart, they linked in your documentation directly to your um to via strips so on the cloud product. Let's try true so I pressed the ps4 and I directly redirected to um my cluster- got ps12 data.

A

B

This is awesome.

B

So we've only maestro: that's that okay.

B

Yeah we get also information about the usage of all our applications about the cpus.

A

Is there kind of a way to use these views and create own dashboards? I would assume.

A

That, maybe not I'm.

B

D

A

Grafana dashboards or something like that.

A

This is not a dsl anymore. That's a programming language yeah.

B

This is what I mean like when you're writing, something like. Oh, this is like not straightforward, but.

A

It's I think so, like I have a python background. um I think this. This could be really like appealing to developers to really like extract what you, what you need to do, or not not just developers, probably also data scientists and machine learning, engineers and others.

B

Let's check the dots if you can export it into another visualization, so.

A

Let's see pixie export pdf, I think that's probably an enterprise feature but exporting data into png or iframe or something.

A

Integrations and alerts graphanapixie plugin.

A

Or you can use pixel as a data source in grafana, interesting.

D

B

Well, then, you would use draft for that visualize it in a different data source.

A

A

At a pixie dashboard.

A

Oh somehow, you can also generate dashboards from pixie scripts.

C

A

Or at least they are like available as json to be imported into grafana.

B

um ah Here we can see likely things.

D

A

D

A

Like this link, which is in the chat now, you can kind of run a script on the pixi grafana data source plugin, and then you get some graphs.

B

Yeah, but you need to decide which kind of which visualization you used before so in terms of like when it's slightly a diagram, you probably need to know if it should be a heat map or something else or a table.

A

Yeah, I I think it's like this is just the query parts and then you kind of select what.

B

A

The data should be shown or what other queries you probably want to group or correlate with.

D

B

You can also export all the data as open telemetry.

A

Yep, that's in the menu on the left hand. Side is the next point.

A

Yeah deploying an open, telemetry collector in your cluster is rather easy using the tele, uh the open telemetric operator. It still says it's unstable, but for testing it kind of works.

B

Yeah we and eks supports it. I think out of watts right now.

A

Aws has a distribution of open telemetry. A distribution is kind of a a package, binary version which also has more enrichment with metadata or plugins.

B

Distro for open territory.

A

B

I know we use it.

A

Oh, that's interesting. It's natively integrated, um but yeah. I remember, I think michelle hausen plus shared that a while ago. um They also have like an empty dog's website where they are sharing integrations.

B

Okay, this is slightly normal thing.

B

Now and then you can send it to your data source entry.

B

Yeah but that's like really cool, so we never instrumented any um any application and we see all the data in there and then you can do a lot of more fancy stuff. In terms of like using this data with machine learning and then optimize your cluster.

A

That will be interesting to see how this evolves.

B

Yeah, it's interesting if it's really community driven or only some people are using it so technically.

B

um So, for example, when you start with ebpf, so I think the first thing that I would do technically is to start adopting cdm from my network.

B

So then I see already over packages so um how I would do tracing probably I would I would not go directly- will open telemetry way. I would probably sing for vendor lock, indirectly to get more features out of it in a small company if you're in a bigger company, for example like we now have of a soft shop so that you have different servers, you need to support different profiles. It's probably worth to look into pixies so that you can reduce the burden of every application. Team needs to instrument their applications, um yeah and.

D

A

B

How resource hungry is this so.

A

Yeah scaling and performance and resource like what is like the cost of running it in your kubernetes cluster would be interesting in comparison, um and the other thing is like auto. Instrumentation sounds super super nice and super easy, but it's like there are like best practices around for different languages, and it could be that exactly the language your team is writing. Software in is not fully supported, or they are like problems of matching uh metrics and syscalls to the code and doing things. So I think.

B

Yeah automation so yeah. That's mostly what people not like to tend to do.

A

It's um it could be like a problem if you're looking like into java, which, where the jvm is kind of made for auto instrumentation, it's easier.

A

um But I'm also like looking forward because currently I'm seeing a lot of startups um in the the observability and evpf space, they are using ebpf to you to auto instrumentation of code, um and it will be interesting how far the competition and innovation is going, but yeah, pixie, um psyllium, um probably, and and and also tetragona, probably the the platforms to get started with. I would say um also like security related falco and tracy and.

A

There are many like tools to get started with, I would say, like getting getting to know the environment these tools help, um but in the end like, if you need like to make customizations or if you want to see if it's compliant with specific iso certificates, certifications and whatnot um understanding, ebpf and how it works in the background is also required. Yeah.

B

But this is not in terms of for the compliance it's not required to understand fully. You need only to know how the data goes in. However, it goes also.

A

Yeah, I think it could be a an interesting, like fact how the data is stored and what are like the potential attack vectors or something like that.

B

Yeah I'm checking now if uh pixy is already on the open, apm. Oh interesting I'll, probably.

B

D

I would see it also as an apm.

A

um Partially, I think we I'm not sure if, like apm, is still the right term, um because every like big window, I moved from apm to observability. Some did a rebranding, some provided real um ways to like do observability, because I see nagios on that list here for dashboarding.

B

A

Doesn't feel right.

B

Yeah actually you're right.

A

I don't know it's I I remember that page. I think it's more like um it's more monitoring focused.

A

You know. On the other side, there is sipkin.

A

I think it's open source, so we can kind of contribute. Yeah.

B

Probably improve the documentation so and simplify.

D

The uk yeah.

A

No, I mean uh open, apm is open source as well.

B

Right, yep, yeah.

A

But yeah, of course, pixie um the things we kind of encountered now I learned now uh could also be. Could.

B

Also be contributed uh in the end without the storage problem, it was quite easy to deploy and we could see a lot of data in there already. It would be no.

D

B

Interesting if we can also host it for cloud stuff for ourselves that we never would leave the data out there and probably to check because they say they don't store the data, um I tried not sure how it works. So that's like a missing point, but they say they don't do it. So if the.

D

B

Is in your infrastructure, it would never leave the system.

A

I think it needs kind of for updating or for life cycle management. The cloud is needed, but I don't think they're. Oh there, it's like it's optional, to store the data in the cloud, but then it's like a paid tier or something.

B

Since they really say no customer data, so outside you're, not work and say, what's the agenda.

A

Yeah, potentially it's like a um a usage, ping or service that allows them to like track things and also um provide you the ability to upgrade when, for example, a security release comes up or something yeah. um But I would say, since it's like a an open source project in the cncf foundation or cncf umbrella, um it shouldn't like have the requirement to phone home. You could run it in your cluster and have no internet and theoretically, it should be working with um using your own container registry proxy or something like that.

B

A

A

I think psyllium works the same way you can. You can deploy it in an offline environment. Theoretically, if you're providing kind of access to the container images and and package dependencies which which might be installed.

B

A

um I think the the easiest way is to sniff all the traffic um and to see where uh what the communication is with uh the pixie cloud.

B

Another option is um to ask for people in an issue.

C

Maybe it would also be an option to to try to deploy pixie into a mini cube onto minicube cluster and turn off the internet. They.

B

C

There is a limitation for a mini cube uh for mac m1 series. It does not work currently.

B

Yeah, you see oh yeah.

C

There is also such.

B

Means you don't have any internet connection yeah.

A

B

C

Yeah but would be uh same thing uh like trapping it in a mini, cube cluster. I think uh if minicube runs on your local machine.

A

Oh, it's cool that they actually like document the errors you're seeing.

A

Oh chrome, devtools console okay.

A

Yeah, but in the end it it works, but it's probably not the cleanest solution so um or it. It requires more maintenance uh rather than just using what is already there in the sauce free platform.

A

And from a from a product perspective, it's kind of logical to to say: hey, we kind of offer this for free and if you want to use more here, you go because.

B

And I think this.

A

Is this is also the reason why we are seeing like the the chat button or something like that and.

B

I think it's not only a derivation or like um an incubation proto project of neurotic indians, so since the neural probably has a lot of interest to adopt it, mostly in the white for their own system and for the enterprise systems,.

A

Yeah definitely I the project itself like if this is only like pixie has 50 contributors. Yes, you thanks.

A

So it's pretty predictive and I think so. Pixylapse.Ai probably there is like additional services like uh machine learning and other things on top, which are yeah, which are like um anomaly, detection or specific.

A

What is called a iops at the moment or has been called like assisted tips and tricks, or remediation or code patches or whatever, um which then like originates from from an enterprise feature.

B

Yeah, I guess so now, but it was like recruit shop to see how.

A

It works and how easy it works.

B

In the end, so nothing really to drop here.

A

And the the cool thing is the pixie blog um also has nice articles to learn about, because yesterday I read an article about uh monitoring, grpc connections with eppf, because grpc is kind of everywhere, where you have microservices for communication and it's not the easiest to debug, I would say- and there is like there's an article on the pixie blog as well. um I will be adding this to my ops, indef dot newsletter which isn't published yet because I was running out of time, but it will be published soon.

D

A

Okay, great anything else we missed.

A

B

Guess not not deployed, or we.

A

Should don't don't? Oh um let me let me share my screen. um Be a pixie. Now, not I don't know.

A

There's like the community page getting involved, there's a select channel. There is a meetup twitter, the repositories and the camera.

B

Oh, why can they not be in the current selection? I need to register okay.

A

Do we need to register.

B

Yeah, it's a announce like it's like, I think, not very.

A

B

Then I'm again I'm coming back to my problem. It's like that. I'm in 20 work space or something like that.

A

Yeah, I have the same problem, um but I I have most of them muted, so yeah.

B

That's but then why should? Why? Should I be there, so I mean.

A

In the end, if you care about the community and the project, you probably will be active in there yeah. So this is something like um probably shouldn't be spoiling something, um but I'm I'm kind of active in in the cnc. No, I'm not active in anyone, but I have me I'm trying to be as active as possible, but the problem is synchronous. Chat is killing my time, so I'm I prefer uh async communication, but this is the twitter account.

A

Doing some free advertisements now um it needs some followers. Actually. So, oh, and this is by the way, the reason why I found the article.

B

D

B

Talking about uh about crafter.

A

Probably um enterprise offering pixie support interesting.

A

Okay, it's pixie labs is a community offering by new relic. Now I got that right. Okay, there was an.

B

B

Mentioned um one engineer uh was like it's a senior a principal engineer at neural and she's, also a founding engineer of pixie. So I think it was an outro. Oh, they have a lot of videos.

A

Learning videos uh yeah- I do remember talking with someone from pixie after my kubecon talk in valencia and wait kind of chatted around slos and eppf, and what I'm thinking about this. um But since then I wanted to try pixie and now we finally got got got around it. um I think, for some reason it didn't work in c word cloud that was probably my blocker um but yeah. What else?

A

No, I'm uh I'm just saying that everyone needs to click on the get started button and try it out and open a pull request or merge request to help improve the documentation, maybe contribute some pixie scripts. When you have created some for your environment, yeah and looking forward to what else we can learn in the future,.

A

Yeah then, thanks for preparing the cluster and they were and like doing some live debugging in the beginning, the only class.

B

Every time like dividing something.

A

Yeah, but it's I think it's it's a great way to follow along. So when someone, how someone approaches a problem and like different ways to hey, I'm not googling for this, but yes, you probably could be googling, no, it's a different way and so on. I think it's uh authentic and uh folks.

D

A

Learn in the future.

A

And also thanks to swen uh for joining today, um like the joke we made in the beginning, without the recording happy new year.

B

C

Same hopefully, we.

B

C

Yeah, it was fun to be part of the contribute coffee again and I learned much about pixie, which is quite nice and interesting tool which I maybe try to adopt in my own projects, because it looks very nice to not having instrumenting all services that are deployed in the cluster.

C

So might be nice thing to learn for the future. um So also special thanks to niklas again for preparing and explaining some of the basics. Around metrics and observability um yeah.

A

And maybe maybe write a blog post about your experience and your learnings.

C

That would be my plan I'll do more uh towards blogging in the future, because I'm uh yeah I tweeted a few weeks ago, I'm currently learning or preparing for a ckad certification. So.

D

C

Will be live streams showing me trying to to manage kubernetes and yeah, learn that and I'm planning to do side projects for those streams.

C

So maybe there will be one or another episode featuring pixie or something like uh observability uh in a yeah kubernetes context.

A

Cool, let me know if I can amplify um or like encourage folks to join you yeah.

C

B

Probably, should you connect to philadelphia because for the I think it has every certification so.

D

B

Cka and for csj.

C

uh Yeah, oh damn so like he's like a prophet.

B

C

B

D

C

Should uh reconnect with him he's.

A

Alright ask you um yeah, currently he's head of noburg because he's on vacation, um but I think he will be back soon.

B

Yeah, I think, in one or two weeks.

C

I am no stress from my side. I have a lot of work to do.

B

But really also in the end would be great to hear about your story about how the c8d was for you. So that's mostly interesting so because it's probably different for people.

C

uh Yeah, uh I read a few blogs about it and uh they all all were like uh yeah, it's a good certification, but it's pretty hard to do it. If you're not like on a structured learning path or something like that um and yeah, I think most people wrote something like uh cks and cka set of certificates are yeah even harder than ckad, uh but it's the most basic one on those three um yeah for me right now, it's yeah. I will I'll. Let myself be surprised, we'll see.

D

C

I think in the end, it would work out.

B

Yeah, don't worry about that.

C

Yeah, it's all about mindset right, yeah,.

B

You need to think that you get a bit radiation, then you would literally radiation.

C

A

C

A

Looking forward to congratulate you well or to say she is my english is not good today um when you pass the exam, that's the thing.

C

I'll, let you know I'll join the next contribute coffee, then. Finally, I did it guys.

A

Just let us know async, don't wait.

C

Yeah sure no problem.

A

And potentially, um depending on where are you located, I don't know exactly.

C

In germany.

A

Which is neonicotinoids, I think, yeah one hour away from me.

B

C

B

uh I'm living in the good old button forest.

C

ah Yeah I heard about that one.

B

Everyone hated about buttons right now, it's mostly close to home, and so I'm.

C

B

In for rural valley,.

C

ah Nice uh is there something like a community in uh nrv too, like a meetup in hamburg, or something like that. I.

B

Don't know if we know, I don't know that someone is organizing, currently development.

D

B

But um I organized in the past, like meetups for cloud native and um meetups, and now it seems like after we know, hitting the pandemic or after the pandemic. That now um more conference is coming, so meetups were also charms. So the promised like finding a place finding speaker like these are the most things that are mostly hard to do, but I guess they will come so probably find some companies that want to host and um and that, I think, will work out.

B

So I still plan to attend no to some aws user groups again um and also conferences to make some advertisement for conferences if you're interested in kubernetes and the community. You should probably join in october that kubernetes community days in unit. So that's like it's.

D

B

But material is also like um it spends uh it's like reachable, it's not like quite expensive, like youtuber, not something like, but because it's a community event, so I can recommend that.

C

Oh yeah, it sounds great uh yeah.

A

Look at it. My my question was also like related to that, like keep an eye out for community or events you want to attend and where we are at and we could potentially meet in person, um and we also discussed doing hybrid meetups, not sure how this exactly would work.

A

But in the end um I think it's like getting together as a group who met two and a half years ago or in the past two and a half years ago um and kind of make make a personal conversation happen, which we enjoyed at container days last week, um but think, like everyone, should be at some point which we should meet in person and.

B

Yeah, probably we made like a two sitting room in the everyone transparently, but oh, we can meet at least for coffee. I can mostly, since I'm I have a good employer so that I can also work from everywhere, so it means I can also easily join our handwork today there and we can start to meet up for beers or financial coffee.

A

B

A

Looking forward to that- and I think I'm trying to convince nico and philip to doing something in rosenheim which is kind of between munich and salzburg and austria. um But we will see about this. So it's like yeah who, who wants to invest what uh and who needs to drive away or.

B

And can, as I say, also.

A

Get a can for me, it's like I'm getting into my car and I'm driving to a place, and probably some hotel or sleeping places in there um and the other way around. So if we're planning on something in nuremberg, you can also couch couches available or we do something in somewhere else. It's just yeah the the community kubernetes community days in munich um and potentially yeah prom con uh is also in munich in november early november, but I'm not sure um I've submitted the talk, let's see, but.

B

The demo see if a ticket limit is um really short, so only 200 tickets or something like that is 300, and then they are already sold out. So it's like when it's most you don't that got pretty fast sought for prom.

A

Probably you can't now the registration will open soon. Okay,.

B

uh Yeah and they make you close to the event so because they know that all the people working there, but it's a true conference. I can recommend them, I'm very interested to monitor and what's going on.

A

Definitely- and I got accepted to speak at another tour which I cannot talk about yet whenever I went um but yeah, I'm I'm looking forward to meet everyone, and if not this year, maybe next year.

B

Egyptian and uh europe.

A

Yeah, this will be in.

D

A

In amsterdam.

A

Yeah, I think that's it like our cfp issues and epics are all public, so this is kind of my my tracking sheet for what's what's going on, I would really need to plan for next year already.

A

Anyhow, thanks for joining today um this time, this is the time where we say goodbye on youtube. Bye ideas see you next month.