Open Telemetry Uncategorized, 14 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-11-14 meeting

Description

Open Telemetry Meeting 1's Personal Meeting Room

A

A

A

B

Hey guys just give me a second: um we get set up.

B

Yep any specific items I want to cover I know we have item or uh Austin's um awesome, spr.

C

C

Yeah, maybe one topic was this: this circle CI postgres, which will run on the docker and if you have, if anyone doesn't have any real reason, then that's one item as well: I think a small Improvement on the locker.

B

Yeah, we'll have to hear that um okay, so we have uh after this PR low hanging, fruit, I, think that should cover most things um Austin, do you want to go ahead and start with your PR.

D

Yeah, um so this has become sort of a mega PR now, but.

D

To catch people up I'm, pretty sure we left it was getting I. Don't so I got larger Runners um last week, I want to say and tried those with the cross-platform builds and the upside is that it did.

D

It was faster, but it was a linear increase in speed like we went from four hours to two hours, so by two exit effectively by 2xing the runners, we got a 2X performance increase which, um unfortunately, probably you know, wasn't really good enough to justify the additional cost because you're charged per minute- um and it was like there was a problem with that build, and you know it took. Oh I, don't know it was like 50 bucks, I think at the per minute rate, but you figure two hours and then you're getting it's.

D

So it's two containers for 100, it's 120, plus I, don't know it still would work out to be like 400 so minutes because of the way that the prorations work and everything, because every single container has you know, gets billed separately either way um that that got undone because of the costs.

D

So I've done two things: one I have an open service desk ticket to see if, because I think we we still want multi-arch builds but I think the way that we're gonna have to get them is by building on the correct, like building on Native architectures right. So if we could get on-demand Runners through some other way um like I know, equinix has like donated has donates servers to um cncf.

D

So maybe we can get like a pool of those and build out our own build infrastructure, but if we can have basically a pool of arm and a pool of x86 Runners and then um Target each independently, then we can build the Manifest independently and push them to the registry. There was a thread in the slack about this.

D

Basically, if you tag the image appropriate, you tag the image with its architecture and then push both independent builds as long as they share everything, but the architecture part of the tag then it'll merge together and when someone pulls it'll pull the correct thing for their architecture, so that that's kind of the multi-arch part in the shorter term.

D

I switched this PR back to just x86 and have done a few things one. um It is actually much faster now because, instead of having to build everything kind of independently we're able to build.

D

um Each image in parallel.

D

I also managed to get uh caching working between checkouts, although it doesn't seem like it works reliably. I have seen it work, though, which is weird, um but there's a GitHub actions cache that the docker.

D

That the docker Builder can use, and it's able to like I, have a run where it was it. Where I saw it loaded intermediate layers from the cache. So it doesn't necessarily seem like it's working right now, because I just triggered a rebuild.

D

uh Unexpected inputs build Arc, so I need to keep working out a little bit, but there should be a way to the problem with uh the inline with the caching. Is it works? Fine for single stage builds, but for multi-stage builds it doesn't work without some build argument being passed to Docker and that's what I've been trying to do.

D

But if we can get that working, then each layer of both a multi and a single stage build will get cached to the GitHub actions cache and then it would only rebuild thing it would only then it would use that cache rather than using, rather than having to rebuild each layers independently, which would solve the slow builds for recompiling, grpc and hotel and stuff like that in like rust, c-sharp. Whatever right, there does appear to also be some interactions with like rust, where it.

E

D

Get cached well that I've been researching, but either way the long and short of it, even without caching. The current multi-state, the current build PR takes us from an hour to about 20 something minutes.

D

um It also has improvements for local build times, because we're pulling in a different Linker for the C plus build called mold that when I ran this on my um Mac Studio I think I built that whole I built the C plus container in like eight minutes.

D

um So I do think it. It cuts things down pretty significantly, I'm, not quite sure what else could be done on the C plus plus side to make it faster, like maybe bazel, would be faster than cmake, but I don't I'm, unfortunately not enough of a like C plus guy to it would take me a bit to like figure out how to convert the cmake stuff into bazel.

D

um I know basil is supposed to be faster, but some of this is just the default. Runners are the default Runners they're, two vcpu they're, seven gigs of RAM, like they're, only ever gonna that that's all they're ever gonna, be until we get bigger Runners, um but I do think that what we have now is probably the best we're gonna get for now with this PR.

D

If I can get this inline caching working and then this would actually take intermediate builds down to probably seconds, um or at least minutes for the like a couple like a handful of minutes for every PR, because it would only have to build changed code so they'll. Basically, the last couple steps in a build.

C

Yeah, maybe maybe a comment there, because I was also running it now in the morning and I. Don't know if you saw the saw the comments, but for me, I was also into seven eight minutes category to build the C plus plus service, but somehow the mult link or didn't help at the end kind of that I think I spent like or it took, I don't know, 160 seconds to get clone and yeah.

D

A

You can do.

D

C

Yeah, it became kind of negative.

D

um I mean we can take it out like it linking isn't. Probably the biggest part I did notice.

D

um Grpc seems to do a lot of linking Hotel, not so much that's where I saw the biggest improvements but yeah. It's like a five minute install so.

A

C

And maybe it's also because of the I don't know network speed, maybe.

D

C

D

Amazing, you would only you would only see the mo the mold improvements on Currency Service.

D

Because I didn't change.

C

It on feature flag or shipping, yeah I tried also the other ones and with.

D

C

Changed but actually the last one, the shipping service, I I, looked at the chains and didn't really understand, but it's like it went for some reason. Couple of seconds faster, but the pizza flag, I got.

A

C

Little bit slower a.

D

Shipping service.

D

The shipping service was just um there was something with relative links and the build context that I actually don't even know. If those were.

D

Yeah, because the only change of the shipping service was the build context, was not getting set properly, so I had to change some of it. um One other option that might help is- and we could probably do this on a separate PR but I think someone mentioned I. Think someone had a PR for this or it's already. There is compiling the Proto's independently. If that's possible,.

D

I don't know, but I I feel like that, wouldn't necessarily improve things that would just move stuff around, like you're still paying the cost you're just doing it somewhere else. Unless.

D

Unless we compile protos and then check in the compiled protos, if we don't have to pay the overhead for building the produce every time, then that would probably make things faster. But then we would need more complicated, build logic to see when those were recompiled or when those were changed to recompile them.

B

Yeah, try that actually, but in when, for C plus plus service, that.

E

Won't work because.

C

The SDK had to be recompiled.

B

First, C plus plus.

F

Yeah so I gave up on that idea. Actually, okay.

D

Well, um I can take out the mold stuff, I can and just I can take out the mold stuff. I can try to get this build ARG.

D

This um inline caching working I do think that if we do that, like that's a significant enough Improvement in build times like I mean because that's over having it um that it's worth doing, it's worth doing the new GitHub actions, um especially because that sets us up for if we can get dedicated, I think the actual path forward has to be like a dedicated arm Runner, because if we can get a dedicated arm Runner and then we can say, Okay build x86 on this set of Runners and build arm on that set of Runners.

D

Then that solves the that that fixes these problems um of multi-arch at least.

D

B

I still get improvements overall, just to like cut it in a third, essentially through two thirds with the good good initial swing.

A

B

About those runners or I'm guessing that's not an Hotel organizational thing.

D

It's not I I've opened a I've opened an issue um to do that.

D

I guess we'll go back to this see if I nine I've already opened a service desk ticket to ask about, can we get.

D

um Access to that stuff, the downside really is gonna, be like it's just going to be stuff that we're going to be responsible for managing because we'll have to like set up. You know.

D

Grips I would guess to they would hit a set. We would have to have like a pool. um Well, actually I guess you could have like. Maybe we can just get like one big x86 machine, one big arm machine or some like have a machine that can spawn new machines.

D

um The other open question I think for this is I would assume we want to if it's still going to take 20 minutes or whatever we probably don't want to run it I. If we get caching to work, do we want to run Builds on each PR.

B

Oud um each PR would be tough at 20 minutes, I'd say no, but.

D

If we got caching to work- and it takes like a minute.

B

Oh then, um that's that's a different question: I'd open it up to the group I'm, not sure on that one.

B

um Any opinion Giuliano, maybe Josh.

G

I mean I think that sounds great I felt bad. Every time, I triggered a new 20-minute build because I had a typo in in one of my br's.

D

Yeah I mean if the inline layer, caching stuff works, then I, like certainly nobody's told me that we I mean for the free ones, at least that there's I I, don't see anything about limits like there.

B

Might be yeah they're, not usage.

D

I mean there might be like a currency limit, but it seems like it's maybe Project based, um but even then, like I'm I'm I feel like we would get I mean I. Guess. The two things in my mind is if we did build on PR, then that gives us the ability to run the images after they're built and do like actual if we want to do like integration testing or something that would actually unlock unblock that.

D

It would also make sure that you know there's no obviously I would assume people aren't making PRS that don't build, but this would give us more confidence.

E

Yeah that actually happened uh a couple of times already that we got PR, that the service didn't build.

G

Is it possible? Are there other options like? Is it possible to only do it after, like it appear, has been approved or something like that.

D

Strong Maybe I.

E

There there is a thing in GitHub that whenever is a new, uh uh a new member sending a PR, someone needs to approve uh the the GitHub actions to yeah.

D

E

A proven run, I, don't know if that's the maintainer, that is configurable. If it is, then that would be awesome.

D

Yeah um I know there's let me look at the thing real quick. There is pull request on pull requests- branches Main.

D

um There is conditional logic in here, so you might be able to do something. I think we might actually it might be. We might need to factor it out a little bit, but there should probably be some way to do run this check on approval.

D

But it it feels like this is maybe the best we're gonna get for now.

B

Yeah makes sense to me: well, let's see if we get caption working and then we can talk about maybe implementing some sort of gated um build for PR because it does seem like it might be risky to do it for every like commit or something like that. We just have all these builds being triggered.

B

E

You said we are now I I, don't mind, um pulling the pr and building it locally. Every time I need to to test a.

G

E

Oud, um if we get like a PR with a lot of services, changing that would be cumbersome, but I think we always tend to go to one service per PR or like minimal changes in the pr and.

A

E

Then I mean only whenever is the C plus plus that we need to spend more time, but all the others are kind to build locally. Whenever we we are validating the pr, so I mean it is an improvement, but yeah I think we. We have um big things to solve. First.

D

Yeah I, just it's I mean it really just is like the rust and the C plus image.

D

Like and really I would maybe argue it's just the.

E

You honestly don't know why rust is taking too long.

D

Same thing as building grpc like building it faster, um but.

B

We want to potentially consider replacing grpc I know that would be kind of a bigger swing, but if we limited ourselves to one or two kind of select items, I think that would reduce our scope a little bit as long as we chose the right ones.

D

I mean that's tough, because you're.

D

I mean it takes us back to the original thing, which is.

D

What is too long like well, I guess, there's two questions, one. What is too long for a build like on.

D

Average or sort of like low, like on average Dev laptop. What is too long right is our Target 10 minutes. Five.

D

um I mean I, guess yeah in my mind, that's the the first question should be like what is the actual time we are. What is the amount of time in minutes that we think we need to hit and then kind of reverse from there? Because if it's like again, if it's five minutes, then okay, you know you can. Probably you can see a world where we get the where we go with sort of the intermediate build stages and or publish intermediates.

D

C

hmm And now we are already in the in the 15 minutes range and I. Think that's a that's quite good in a sense. But but we have a risk that if we had many more services, then I don't know if someone gets out of memory or if you.

A

C

To use just all the resources to just run this demo.

G

Also on a cutting edge, Apple laptop right, like not everyone's, using that yeah.

D

I guess that is one thing to keep in mind or that's one thing to consider at least like just Theory crafting moving away from grpc and towards just an HTTP API that we def would give us more, um not saying we should do this, but I'm saying.

D

If we did do this, then you know what would that change in terms of how we did things we would probably replace you know we would probably use like Swagger or something like we could use code gen to sort of create to create the client server stubs for everything, but it would also reduce the weight of each service because we wouldn't have grpc we wouldn't have.

D

um It would give us more Headroom everywhere. I guess is basically my point, so we could add more services. You know.

A

E

I think also, another valid point uh is that whenever we use HTTP uh and we just use kubernetes Services, the load balancing between those Services happen better, because grpc has like a H I forgot, the name but like um whenever, like long living connection. So whenever the service is connected, it will use the same endpoint all the time yeah and uh with HTTP that gets a new, a new connection, every every request. What is nicer to Showcase like load, balancing of kubernetes service itself?

E

I, don't know if anyone is doing that but yeah. It's just a point.

D

Yeah, like load balancing HTTP, is easier than grpc in general.

E

D

um Also, if we wanted to like add service mesh or something right like that same sort of calculus applies.

G

For the purposes of demo, wouldn't it be valuable to have both.

D

C

And we kind of have both and I was also thinking that they are to replace the Currency Service grpc with http that we would get rid of this longest. Beetle time service.

E

Yeah, that's actually a good point: I mean we can replace just the Currency Service uh grpc keep some keep the others, but then we remove the the biggest one and we still have grpc in the demo. What is nice I, I I'm against removing like everything all the grpc calls from from the from the diagram.

D

I would also say: I don't want to remove all of them, but I do.

D

Right now, you know, grpc is sort of the primary way that we, the primary rpcs and I'm wondering if maybe it should be, not the primary like.

D

I mean- or maybe this might be like a 2.0 thing right, maybe the maybe the the right now version of this is we factor out grpc from the two places it's slow, which is I, believe shipping.

D

And turn this into just http apis and then on a longer timeline.

D

Think about can we make? Can we make like a a segment of the application use grpc like that? That makes sense for grpc right, like I, like sort of a logical sub-segment of it rather than everything using grpc?

D

um Would that also be more reflective of sort of real world usage.

G

I think so I agree with all of that.

F

I I wanna I wanna speak up in support of a shipping service, specifically uh going more towards HTTP, because right now we have kind of a lot of corner cases of how you have to handle context um guards and how you handle asynchronous uh rust like threads and how you keep the the span alive over different um procedure. Calls. That would be easier if you were using just HTTP um the way that the runtime is built right now you have to do like async context, that's separate, but it still keeps that span.

F

So you can't use it back in the original context, because once you've moved it, it can't really move back.

F

um It works better when we have to call the uh the PHP service than it does when we have to call the other grpc service, for example, I, I, guess I'm. Just speaking up and saying it will probably reduce build time to not have to do grpc, because tonic is by far the biggest dependency um outside of actually compiling the grpc server, and it would, it would like it would be worth like investigating. If we just took it out, how long is the build? Is it dramatically better?

F

Okay, maybe it makes sense to keep it as HTTP. Only then.

D

I, like this I, think this.

D

Yeah I think that solves multiple problems and gets us to a better place in terms of like usability for end users.

C

D

Mean the other thing to keep in mind is like: if we can, if we do get the, if we can get wall, clock builds down low enough, um then we can turn back on multi-arch through the Free Runners, because what kills multi-arch and the Free Runners is anything like the G is really just it's the same stuff. It's like out of memory errors due to trying to parallelize um simulation I mean that would still probably put us at like a like. Well actually, no, we would just do it on release.

D

We wouldn't build um between caching and between caching and restricting multi-arch to release builds I. Think we could probably get release, builds in the 20 to 30 minute range emulating on the Free Runners.

D

This is Theory crafting on my part, but I think that would be you're.

F

Gonna send me there Carter, if you want.

B

D

If and if the cost is just removing grpc from two Services, then I think that's probably worth it, because that would dramatically improve performance for people pulling locally on Apple silicon.

F

I mean I'll be able to testify in the issue because I am running M1, so if it works a lot better, I can I can post the timing that I get all.

D

B

Well, you're not coming up Gary, but I, just added you in it! So um speaking this on yourself.

B

A

B

About the Currency Service too sorry Juliana I'm interrupt you go ahead. No.

E

No I I started after you. Sorry uh just one thing: when we also need to make sure that the caller is changing.

A

E

F

Absolutely absolutely I I wouldn't want to break the actual incoming call.

E

Cool so yeah that wouldn't change much so the front end would still have a couple of grpc calls. We would get rid of the the checkout grpc and also checkout would need to change the call for front-end and also payment service, because it's doing a outgoing call.

E

And shipping service- maybe change the this call then, but maybe this is a bit tricky. The the call to feature flag service because is.

F

That live right now. Do we have a call to Future flag service from Shipping.

E

I'm not sure what! Because of you that implemented, that with.

F

uh Didn't write that no no.

E

F

A

F

Know if it's actually written it might have been something that we thought was written.

E

Yeah I think, uh if wasn't, you was Pierre. Okay, maybe maybe you.

B

Have a feeling Pierre probably did something around that because he was making some feature fly changes for a while.

E

G

Yeah I'm not sure I'm right but I, think it's the product catalog service, not the shipping service.

F

I'll take a look real, quick.

B

Yeah, probably worth updating.

B

Okay cool now this would be um two good initial items, so shipping service and currency service will be replaced, that's kind of hard, too um top of mind or top performance Dylan's, and then we can reassess post uh removal of these two and see what the build time improvements look like and then maybe schedule a more comprehensive grpc refactor for v2 um and start adding new features off of that or before that um awesome. uh Well, Miko I think we have a little bit of time for your items so something about the the postgres.

B

Was it the issue or is it something specifically? You know like a CI check that we're not using.

C

Yeah, it was uh yeah small topic just that we are using different postgres images in the for the docker and for kubernetes, and it looks to me that it's just a small mistake but um but yeah and if no one knows what would be the reason to use different limits. Then, because in the docker we use the circle CI version of postgres, which is really large, like uncompressed, it's more than two gigs and seems completely unnecessary. So I can try to align it with the kubernetes helm, chart image and.

D

I, don't I, don't know why we would be using the that one so yeah.

A

I, don't think it comes with anything.

D

B

I'm, supportive of that, so you catch me, go yeah, um let's see so I'm, not sure if we wanted to take a look at our issues. I think we're we're about out of time today, but um if any bugs uh or top of mind so there's some traces.

B

um Some cart view issues um and then the front end container. So hopefully, before we release 1.1, we can get some of these smaller items uh of fixed um and then move on.

B

And then I think this is still like the multi-arch issue, right Austin, yeah.

D

I would really like to figure out why that happens because I mean that's the other other option here is yeah to understand why those failures occur. Then.

B

Yeah I have noticed the AMD tag locally, but it's only been on one service. For me, I'll have to double check which one it is I'm, not sure. If it's a feature flag, if I'm.

D

On an M2, Max I might make it anything like it well, anything that gets pulled should have the AMD tag. um I just I, it's weird to me because it.

D

I don't understand, why can you scroll up? Is there a log from.

D

Yeah, like I, don't understand why.

D

The future fight service would be core dumping when, like what would be causing a seg fault.

B

Foreign thing, I can think is like some sort of memory access issue, but.

C

um And you can see the warning from the docker that if you run a different architecture on the Mac M1, that kind of that be careful. There might be issues.

D

Yeah I mean I I, guess something in the way that it's doing it's running the emulation layer, but still it just it's weird.

B

Yeah we're gonna look at it more later, um but if anyone has any ideas or anyone Max and is willing to investigate, please uh please help us out.

G

I can investigate a little bit, but I might get bored.

B

Well, only if you get bored enjoy your life outside.

B

um But awesome yeah I think we have around five minutes left. So does anyone have any specific items or going to give this time back.

E

Quick one: uh we have a open pull request on uh changing logs for the Currency, Service, I, think sorry and yeah. So, basically before we had this single line here that uh the person is is mentioning.

A

E

Now we have this bunch of stuff and I tested this PR locally and basically all the logs were just the the card service like I I got one or two lines in between, but that was massive. uh Do we need all of that or I mean.

B

I'd say probably not, it says it's fixing like not having kind of output issues on a connection happening, but it didn't seem like the information in this PR we're fully answering like what wph was looking for. So um I.

D

Mean is it using what's using the oh, my hotel, this is a c-sharp thing.

B

It's probably using ilogger there. It goes.

B

D

I mean we should I mean we should use the Hotel logging.

B

Stuff by log we want produced as a log like and that being stored not being stored or like being presented on the console necessarily um in my mind, but that's just my opinion.

D

Yeah I mean I think that this should be I think we need to have a if someone wants to like come up with a design dock for, like our actual logging story, because I do think we should have one of those pretty soon.

B

Yeah I'm going to create an issue tied to it um if anyone's specifically interested at um feel free to assign yourself.

B

I think it's a placeholder for now and give it a to tag.

B

Okay, um well yeah, so if you're interested in helping us out in a logging story and I just missed all that, so it's cool um yeah feel free to assign yourself. If you see any bug, issues feel free to take those on too um and yeah I think we should be good for today, thanks for joining everyone, all.

C

D

You um thanks everyone bye.

D