GitLab Backend Pairing, 9 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: BE Pairing EMEA/APAC - 2023-08-09

Description

Today we discussed authentication options for remote development and time precision in RSpec tests.

Remote development auth issue: https://gitlab.com/gitlab-org/gitlab/-/issues/421289

MR with time precision issue:https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126530#note_1496791930

A

Somewhere, which you don't have to worry code Let.

B

A

Refresh, let me restart that, so this problem is related to uh remote development. Remote development is essentially where we are trying to move your development environment away from your local machines to something somewhere running in your Cloud, where you don't have to worry about anything, uh there's a lot of hand waving in that statement, but that's the gist of it. uh We recently released a beta version of it somewhere around I, think May or June I, keep forgetting um where we are only allowing you to create workspaces.

A

uh Workspaces are term of say, you know, developer environment, a workspace for a public project and right now we are working on enabling that for a private project as well- and this is where we are right now, so we've done a spike on it, where everything's working and uh we have a spike uh fully working. The only thing over to not over here is we are using personal access, tokens scope to our user uh and we are injecting that personal access token into the workspace.

A

So the project is cloned at the start of the workspace, and once you start using the workspace, you can do any operation related to it, because it's just a gitlab token and git operations automatically work out of the box. I'll share my screen and show you what I mean by that uh we've received feedback from uh quite a few people that we should try to move away from apart, uh primarily because it's a pact, it's long-lived, you don't want to have a pad.

A

uh Can you have something more ephemeral right uh and that's the context um and the benefit of that is obviously additional security. It gives us an opportunity to automatically rotate those credentials every one hour, two hour and inject those into the workspace again so just in because of workspace, a user can install any software inside of workspace right. So the security profile is more uh sensitive over there. There can be any malicious program that is leaking that token somehow.

A

So, if we have this way of rotating these secrets every couple of hours, it kind of improves our security profile as well. So that's the context uh before I share my screen and just show you what that looks like any question so far,.

C

uh I have a very basic question about what remote development is, which is like how, in my mind, remote development in the web. Ide are like the same thing.

A

C

Are they different or what additional information be helpful to have for remote developments?.

A

Yes, so the only difference uh from a product perspective- you don't want it to be any different from an engineering perspective uh uh like a product positioning, is that okay, the remote development is sort of an extension of web ID, but from an engineering perspective, it's different because in the webrid you cannot run any commands, whereas in remote development you will be able to spin up a terminal run, any commands that you want uh and anything you want to do right and the web ID is purely running in the browser as a right now, whereas uh for a model I mean it is not running in the browser.

A

It's it's consuming some resources somewhere in the cloud. We are basically spinning up a kubernetes cluster provisioning pause and everything over there and syncing everything back and everything. So it's consuming some resources uh somewhere, whereas in web ID, it's your own local machine and the browser inside of that.

C

Okay, so yeah I've, okay, that makes sense.

A

Any other questions.

A

All right I'll take that as a no with that I'll and famously and I'll show you what I mean by all of this right. uh So I have my GDK running and everything. uh I have a private project. Over here I've learned the gitlab UI private product made and push it on. My GDK I create a new workspace from I select the project I select before I get into that. Let me spend two minutes on what this architecture looks like. That will set some context on okay. This looks scary. It is not trust me.

A

uh This essentially is gitlab multiple components of gitlab nginx rails, caches, kubernetes agent, server, postgres gitly. This is our kubernetes cluster and we already have a component in gitlab called the gitlab agent for kubernetes.

A

That is like our way of communicating with kubernetes um and the reason why we need an agent over there is because- and this agent actually communicates over grpc and there's a reason for it is because, uh if the communication was the other way around from rails to this agent, uh we would hit some firewall rules, because uh uh the network, uh Insider customers organization is always usually very strict and blocked. Any incoming traffic has more restrictions than any outgoing traffic.

A

So if we have a component residing inside their Network and that initiates the conversation, it is much more uh open and.

B

That is why we have an agent.

A

Over here, which communicates strategy communication over grpcs with cash cash forwards it to rails, and uh uh so.

B

Let me tell you how.

A

This entire thing works right uh when I do this, when I create a workspace, this is the part I'm working over here, I communicate with gitlab rails, I, say: hi, hey, I need a new workspace, expose it into the postgres database and that's the end of that transaction. uh Now what happens is after, uh like at regular intervals, the agent will ping cash like uh okay ping cash, which will forward it to rails. Asking for information. Hey, do you have any new information for me and gitlab rails will be like hey this new.

A

uh You know this user has requested a new workspace. These are all the information uh go about and create these kubernetes resources. For, for this particular workspace, it will receive that information. It will communicate with the kubernetes API server and create those kubernetes resources, and once those resources are created it will uh you know once they are ready, it will Echo back its status and it will go back its status, but it will get persisted in postgres, which is reflected to us over here right. So it is just getting ready.

A

Let's just wait for it once.

B

A

Is ready, uh you can see the status away right. It is starting right now. I already have a workspace which is already running.

C

So you just to clarify you just said there there we don't make requests from Taz to the agent, but then, when you click that button. Yes, how do we know to spin up a kubernetes? Yes,.

A

So when I click that button, what happens is rails just pushes that information into postgres and that's the end of it? uh My agent is constantly pinking cash. That do you have any new information for me right uh and that information uh request is forwarded to so cash is like a thin proxy over here. It essentially goes to rails rails, then queries the database. Is there any new information? And yes, there is, and it just sends it over to agent agent- creates those kubernetes resources and what it does is.

A

It also keeps a watch inside kubernetes. What has the status of these resources changed if it has changed? Okay in the next poll that I will initiate I will also send over that information, so that information then gets updated into the database and with that, the status of my workspace has changed to running if I open this.

A

Url, it will open up uh vs code. The server of this vs code is injected into the workspace itself. If we like within the workspace of vs code server, is running and the client as well- and we are just connecting to that over here- and we have a fully working terminal over.

B

Here right now, this.

A

Is just the context we've not even come to the problem, any question so far.

A

Okay, I'll leave that as a no.

A

So what happens when so? What happens when this is spun up right? The first thing that it does is it actually clones up uh clones? The repository this is the git Library private repository that gets cloned and once it gets cloned, uh the other parts get started and once it is fully ready, I get a URL I browse the URL. We have a vs code, server running and accessible from the browser, uh because I have my token injected right now.

A

I can just make any changes over here and uh you know just use this UI to commit and to push.

A

And it will transparently do that git operation for me, because everything has been configured I've configured, my gitask pass and everything like everything has been properly configured and if I look at my project over here, we can see the latest comment right.

A

So this is what I have working as far as the spike is concerned. Now this is where the issue that we I need some help with, which is this one right now I'm using personal access tokens I want to explore ways. I can remove that dependency and some of the feedback that we've got is. Can we use some any form of jwd right um and I'm sure we can? The thing is I.

A

Let me say: I am not really a ruby rails engineer, so uh this is where uh I'm stuck uh and authentication and authorization is not my strong suit. uh So I know it's possible with jwds I. Just don't know how to go about it. I did some digging around the rails, monolith I found. We have a JWT token class, uh but I could not uh get it working, and this is what I need help with.

C

Nice, okay um and one question I had for you are you're not on the same team as tan Lee. Are you Stan tan? Okay, um because there was a feature that I worked on with him or he really worked on it and I. Just looked at it. I should say uh around jwts for code suggestions in the web, so he he had a similar kind of pickle. How do I do authentication the challenge? There was a little bit different.

C

It was less a concern of um using a pat like a pad not being secure right, because it's a long live token. It gives you access to more than you probably need for git operations. uh In that case, the issue was more. They wanted a self-contained token because um with the JWT it contains information about the user, and you can add claims right.

C

So you can have information about which groups this user is a member of and other things like that, which means you can make decisions by just looking at the token about what they can do, whereas with a pat you don't know, if that's valid until you make the request to get.

A

C

Like you don't know, if it's been revoked, you don't know, I mean I, guess maybe you could know that it's going to expire.

A

C

But if somebody revokes it, you don't know that until you make the API request, whereas jwts are short-lived, which is I, think one reason why you're interested in using them for this, um and so you just as long as it's valid. uh You assume that the information contained within it is the right.

A

Yes and the reason why the jwd is also kind of appealing is because, like you mentioned, you can add some additional metadata to it right and let me talk about a utopian future world right where I uh I'll only allow pushes to this git repository if it is from within that workspace is right and I can maybe inject some of that metadata into that JWT that uh this jwd should only be valid if the traffic is coming from within a workspace, uh and this is the workspace ID again, there are a lot of uh moving pieces.

A

There are a lot of things that we would need to do, but the point being it opens up a lot of possibilities for us right um and coming to your uh you mentioned about the code. Suggestion part right, I actually checked in with Alpha I think this is the this probably might be a different one. It's related to suggested reviewers.

C

Family, oh yeah, correct.

A

So yeah the only difference this. This is what I when I spoke with Alpo yesterday, um so he in fact pointed me to the same Mr. The only difference over here is they are doing this as part of this is not in the rails monolith. Right this token, the signing of the JWT and everything the only thing that's in the real monolith is the verification of this token. uh This endpoint and what I want, is basically the generation of the token uh as well in the rails.

A

Monolith part uh so I I did I, did not skim through this entire Mr, but I figure out it's a little bit different.

D

Yes see same as you have some background on it: what's the the suggestive review is, there is, is kind of described as being outside of the the yeah the browse long list. What what is it? A completely standalone um web, app or I'm, not sure, like I'm, familiar with the suggested, reviewers kind of top right hand, side, sidebar functionality, but I didn't know that it lived somewhere else.

A

Honestly, neither do I I think this is a part of the gitlab like uh no. This is reviewer.

C

Right not even code, it's um I think this is the suggested reviewers AI feature so I think that piece is outside and.

D

The fact that the Mr is inside it might be a free look at the changes, but.

C

um I think that the end point to generate the um the JWT is in the rails. Monolith is my memory.

D

um So I I hope.

A

D

We've got here should be sufficient for us to to kind of borrow.

A

Right and yes, which class is this, suggested reviewers yeah? This is the one right. Yes, um so I did check this out. This is so essentially it just creates the payload for the token uh access levels and everything and the create service, and uh uh where is it.

A

So it's creating a personal access token. It looks like, but.

D

There's that um I'm just just wondering if it, if there's a JWT, maybe always have a personal access, token embedded in it or.

A

Yeah, this is where my knowledge of entire authentication drops down like a bag of cards.

C

Yeah I believe the um shoot I'm gonna mess this up. If I try to drop them off. My head I do not believe that the JWT always has to have a personal access token associated with it. um I also just realized that I was thinking of a slightly different Mr from the same author. Interestingly enough, um this person loves jwts.

C

um uh So I can try to take that up this, the one I was looking for was for code suggestions, not suggested reviewers they're, both AI features so they're using these external services um and I can update you all I mean one question I had is: um have you considered? No, this wouldn't work. I was gonna say, have you considered using oauth as a flow, but you can't use an oauth token to do a get push.

A

C

Can- and this is where.

A

uh You can so okay, we'll have to take a detour a little bit um over here. What happens is before the workspace gets started right like now. This will be little bit of kubernetes, but let me just show you what it will be.

A

A

Okay, so this is kubernetes, but essentially there's something called as an innate container. That container basically will run before anything else right and we have to clone the project as part of the init container itself, uh and the reason for that is, uh we cannot have the workspace ready like the URL is ready and the user opens it up and then, in the background, uh the project is getting blown right because for a big repository like gitlab, that will probably take five minutes. 10 minutes that significantly impacts the user experience.

A

So what we do is the cloning has to be done. First, only then will we Mark the workspace ready, which means the token available, for it has to be. The token has to be available for or to happen. The user has to do some flow and that token needs to be injected over here right.

A

um We if we were to not do this as init container and just do the other way around, which I just said that could have been possible because I would click this URL over here. This essentially doesn't work by the way, uh because I'm already authenticated it's not uh showing you.

A

It does that a lot, but we are not storing those those tokens anywhere over here right now, uh so one of our ideas was we'll use a pack for cloning and then the port for other operations, um but then I was thinking if a JWT is possible right.

A

Why can't I just use a JWT for everywhere everything right, because uh the JWT can be the provisioning of the jwd can be controlled from the rail side completely, whereas if I were to do a pad plus an oauth right, uh I would need some component inside my kubernetes cluster, which is gitlab. Workspace is proxy, uh which is doing that authentication uh to store those secrets into the kubernetes secrets, and customers are not really happy when some component has access to kubernetes Secrets, they get really touchy about it.

A

uh So I was just trying to avoid that uh eventuality. Does that make sense.

A

I know: I went around quite a few things over here, but trust me, that's the easiest way. I could explain. Yeah.

C

Yeah, no that's good you're, giving us! This is like a good insight into something that I've not known very much about before today. So it's, it's increasing my knowledge of remote development by like 100x.

C

A

So what I was trying to do up till now right, sorry, I could I was essentially uh still trying to play a play around with this particular thing. Essentially, uh jagos was mentioning that and what you see token is essentially a gitlab JWT token, which has some uh those are all the required uh uh Fields set and I was just trying to play around with that in the rails. Console, but I did not get it working.

A

So maybe do you think if it would be a good idea to and I know, there's another issue that we have so uh a real time boxes to maybe next five minutes or 10 minutes and uh well.

C

Mine is not I added that all right, if it's just mine, I, just added that uh as like a function but I feel fine. I feel fine moving forward with just.

D

This yeah I think it's an interesting problem um and if you could potentially go back to the Rouse console and show us kind of what you were digging with or yes.

B

D

To build something in into code, you could but browsed on sales. Quick! Isn't it right a lot quicker than keep recompiling.

A

No bear with my uh uh almost childish knowledge of rails over here.

D

So what I'm I was trying to I shared a blog post era that I haven't really been able to multitask and read it fully understand. It fits relevant um but I'm interested to know which you know do any of our API endpoints. Currently support JWT authentication like what what do we have? I I think we already have a mechanism to generate a JWT to authenticate with gitlab, but what can that JWT actually be used for um I? Don't really know so.

D

You know I I think this is kind of similar to your approach and hopefully everyone's you look for somewhere else in the code, but we already do something along.

B

These lines and.

D

You go from there, so if anyone knows anything that we can already do using the JWT, um I I know, there's there's something to do with CI and hence that blog post as well, but it's probably more for maybe CI to access other systems using their jwts, maybe rather than ours. But.

A

Yep, this is what I was trying to do right, and here we are uh uh so I was just basically doing a general search about jwd's entire code base and I did find, which one was this. There are multiple things to be honest, some of.

D

A

Some of them, don't none of them, make complete sense to me. um I figured out, there's a inbuilt class called jwd and you need to pass in the payload. uh So, okay, the payload looks like jdi is just a random uid, so that replay attacks cannot happen is who is the issuer? In this case, this will be gdk.test when it was issued at what is the expiry and the subject on whom do we want to issue this uh spoken?

A

Who it is related to all right, so I set that as payload and eventually I want to set the access level as well that this project this token will only be valid for maybe write or write a repository or something like that. I create some secret. uh What I realized is you need a SSL key for signing the secret I was not aware of that.

A

I thought a generic random secret would be available, but it looks like that's not the case, uh so if I I think this was the one you know on this, one give me this one: no.

A

A

D

Yeah, that's like the one you were maybe talking about Jesse there, the code suggestions, JWT.

A

D

This is the class.

A

And uh this is what I figured out from looking at the code. Okay I need to pass these things, but after the pointer, I just lost a few things like it's just throwing me I was like.

A

The secret has to be uh uh what do you say an RSA instance. I was under the assumption that that is not a requirement, at least for JWT. You can sign from any Secret.

C

But I think the issue is that we're using so there's like a well-known open, ID configuration endpoint, I'm gonna, put it in the chat and that is or um that is for our existing open, ID connect Integrations, and so when this new JWT Behavior was built, um they kind of piggybacked on top of our existing oidc logic, and they were like okay, we'll just use like the signing key that we already have or the um the jwk that we already have so um I believe. That is why that is in that way.

A

This is that key that you're talking about this is so I. This is just a 32-bit random key. Oh.

C

A

Yeah, uh because this is the same key that we are using for encrypting our secrets when we store it into the database uh under encrypted. uh So again, that's what I thought so I just deleted random call, uh but yeah I'm yeah. This I become a helpless person. After this point.

C

So I mean one thing that I will say is if you wanted so like the route you're taking right now is like you're hand, rolling a JWT right like.

D

C

Not really hand rolling it because you're using an existing library in our code base but you're. Looking at the code and saying oh, we have this library that makes makes it possible to make it to do T. Let me see if I can create one which might be the route you end up going. That is the route that they went for code suggestions because they didn't want to use oauth. There was a very specific reason they couldn't use oauth.

C

In that scenario, I can't even remember what it is at this point um if you did want to piggyback on oauth. This would be a lot easier because we already have the ability for gitlab to be an oidc provider. So somebody goes to the uh the oauth flow, but you get back a GWT with that flow and that JWT includes the information that you want.

C

um More than happy.

A

I think, if that's possible easier to do that's not a problem by any means.

C

Yes, that should be easier, I'm trying. What I was just trying to find is. There was like I, had a crash course in oidc some months ago, because there was an oidc related incident when we updated one of the gems, and so I was like, ah like what even is oidc at gitlab and like how do I.

C

You know investigate, what's broken here and uh so I do somewhere have like very uh beginner friendly instructions, because I wrote them for myself on how to set up oidc uh plus oauth Advocate lab um I hope that I put that in the documentation somewhere, but I I might not have so anyways I, don't want to spend too much time taking for that. But I can help you with that. After this.

A

C

I think what we could do, um if you wanted to just figure this out in real time together, is we could follow the documentation for gitlab as an open, ID connect provider and try to set it up either with your GDK or with just you know, getlab.com or staging.getlab.com, and play around with it that way. I.

A

Already have an oidc set up because uh over here right, this is the one you're talking about so yeah. The reason why we have it set it up is because, like I was mentioning that when the traffic comes to gitler Works business proxy, it will first authenticate and authorize with uh gitlab, whether it's valid or not. So it does an or3 director to gitlab, and so what we do is when setting up remote development, we first create an oauth application and the client ID and the client secret is passed to gitlab.

A

Workspace is proxy and it does that oauth redirect. So we have already have a existing uh application setup. You can probably use that.

C

Yeah I think yeah, we'll have to see if you can um I can't remember which attribute you have to set in order for an oauth app to request. I think there might just be like a specific scope. You need. Oh, it's I think it's profile, yeah, it's a profile. You already have that and open ID okay. So when you get back the response from that flow, I think you should be able to fill out the JWT.

A

But the flow the response would be over here right, because this is initiating the redirect so essentially you're, suggesting from the code base itself do a redirect over there. That's what it's interesting right.

C

um Can you rephrase them so you're saying in gitlab workspaces proxy? You would have the oauth access token in JWT, but you need it somewhere else. I.

A

Need it in the rails, but the reason why I can't have it over here is because uh it's.

A

It's firstly for the traffic to reach over here the works. The user has to browse the URL of the workspace for the URL to be available to the user. The workspace needs to be running in a running state for it to win a running State, the project needs to be cloned or the project needs to be cloned. We need a token.

C

So you have to start with the token.

A

C

A

If I yeah, so maybe in the rails code base itself, I do a redirect and to uh get the token from there, like as part of my code base right, let me show you what I mean by that.

A

uh Over here, I'm creating a personal access token right, I'm thinking should I just Google or redirect and try to fetch a token over you. Instead of creating a personal access token that should work right technically.

D

A

Sense, but wouldn't the user have to uh can I be because the complication like if gitlab is the issuing authority I think we can still work around with it, because otherwise user needs to explicitly authorize. Okay, you can. They can have the token right, but if someone else is the issuing authority, let's say GitHub or Google, then I I. Don't think that would be that, like how do.

B

A

How do I authorize on be of the user itself if I were to do it in the code?

A

Okay, I'm, just I'm, not 100 here, so maybe some of these questions are not framed correctly.

C

No I think you are being clear: I'm I'm, just pausing, because I'm thinking about how we would do that um I mean we have access to all of the live. The oauth library code in the rails, like we are the identity provider and get lab rails, and we have all the oauth codes. So I can't think of a reason why we couldn't just generate the token within gilab rails and like give it to ourselves right.

D

Yeah that that's kind of what I'm feeling we're the ones that have added this, what we can call it like wall or protection or or a kind of process that involves kind of redirecting and authenticating um but yeah, it seems like we should be able to to kind of just skip that step and and go directly to the the bit that we need.

C

D

C

Guessing that one thing, that's probably been confusing as you've looked at this is that um not all of our oauth code lives in our code base. We depend on a lot of external libraries, for our authorization and authentication code and I find that really confusing and I work on it every day. I'm sure it's really confusing. If you don't work it any day, even more so like like what needs to be invoked exactly to create an off access. Token, like we can do that, it's just a matter of figuring out which library is doing it.

A

C

How they do it and then calling it.

A

And I'm, looking at it from someone who's of who's like who's, mainly working code right on it. What is this? What's going on, I have no idea right. I, barely I I can only understand the syntax to a certain extent, after that, like okay, I need to do some more research about this okay. How do we want to proceed over here? Should we then uh drop that oauth idea and just dig into the trenches of is issuing the token directly or.

C

I think so I mean I. Think, based on the limitation, you were just discussing I, don't think the OS path would work because we're not sending a request over to the to kubernetes or the OS access token needs to be in Gala rails and they request it.

C

um I guess one question I have is: how do we know that we trust the you like? How do we know? Which user is making a request to trust them?.

A

So each user is, uh has a one-to-one mapping with I I mean each workspace, has a one-to-one mapping with a user each. So there's a one-to-one mapping between a workspace under user one-to-one mapping between the workspace and the project from which that workspace was created. This might change in the future, uh but uh we have a one-to-one mapping over there. So if you were to access this URL right now, if someone were to access this URL, they will not be able to get it because uh what happens is when you access this URL.

A

It goes from here to the load balancer. It reaches kubernetes from the English controller. It reaches gitlab workspace proxy. It will do auth redirect uh it will get back. Okay, it will authenticate you and as part of authorization it will check. Does this user has access to this particular workspace? It gets the name of this workspace from the URL of the workspace itself.

A

If you look at the URL, this is the name of the workspace itself, so it will do that authorization check and it will fail over there, so you will not be able to access it. So if any traffic that is coming from this instance will be from the user itself as uh from the authenticated user, because for me to access it, I'm first authenticated and authorized that is from within the workspace from outside the workspace. There's, no user interaction like when we are doing that project. Cloning right that happens even before the workspace gets started.

A

So that's more like a service account kind of thing, like there's, no user interaction at all over there is that answer. Your question I feel like.

C

um Yeah no I think that's you're, doing good you're doing great you're doing great, um because.

A

We are, we are a difficult time trying to explain what remotable event is when we are trying to move it. uh We if you were like we are those people who are like building this conspiracy theory. We are connecting 10 dots and uh you know so. Yeah I.

C

Think it's good. You know. I was actually just talking to a friend of mine, who is an engineering manager at GitHub, and you may already know this because you're deep in this problem, she told me that GitHub, you can only do remote development, so engineers at GitHub cannot do local development anymore. This has been true for several years now. Remote development is their only uh means of working on their own code base. I.

A

Think actually dog food in it like they created it and they burned billions of dollars to actually make it to what it is the code spaces right now and we just starting off to compete with them. Yeah.

C

Yeah, it's um well.

D

I I I think the interestingly one of my chaos. This quarter is about trying to start um I, don't know what the word is: monitoring GDK.

D

um So we can see how how stable how reliable Etc is, because we know that we we have issues with TDK and a lot of them are based on everyone having a very, very subtly different development environment. You know I'm on Linux you're, a Mac Europe, 15.1 you're on 15.3, you've got rvm installed, you've got ASDF, installed, etc, etc. So um I haven't done. My team has been working tirelessly on getting um GDK working in in remote development workspaces.

D

um So you may have already had a conversations with Raymond and who's making great progress. We've got customers who have got hugely hugely locked down environments that can't contribute back to gitlab because um they they they can't do a clone, for example, from gitlab.com um and again we can unblock them by being able to provide them remote development, workspaces Etc. So it does solve a lot of problems um other than the fact that we'll have a head of a lot of processing power on our desks. That's going unused, but um you know yeah.

D

Well, that's that's another thing. You know: we've got contributors with eight gig of RAM that are and spin up the GDK Etc. So you know it's it's very exciting um what what the possibilities are.

A

Yeah exactly the problem we're trying to solve, which is different development environments so uh affecting people. We have one more customer who's trying to contribute.

A

Two remote developments is a TV started in January, but they are still trying to get their GDK or working believe it or not. After a point, I just gave up on it and I was like okay. They are not going to get any contributions from them.

C

Wow, that's that's pretty.

D

Disheartening but like like I, say when you consider the fact that they can't even whether it's the same one or Not that we're talking about they can't even do a clown from gitlab.com. You can start to understand some of the challenges.

A

This is yeah exactly. Oh, so I think this is a coming back to this entire JW living right from this blog uh energy I was even looking at it yesterday. This is the class that they are using to the CIA GWT two tokens version two and if I look at this I think this is using the identities that we are talking about right like because we are ourselves issuing authorities uh within the world. They are probably using this.

A

Can we try this one out? The only catch over here is the subject over here is, uh it is linked, I mean scope to a particular project and that particular job, but I want it to be. You know, uh scope to that particular user. I obviously have the user ID.

D

Yeah that'd be nice, that's a problem. It also looks, like maybe I think you were just putting the user ID before yeah, whether this this kind of hint that maybe it should be prefixed in such a way that we know that it's a user ID. But you know it. It's probably not overly important at the moment, but only.

A

If I had a step-by-step guide of how to do this,.

D

Well, it sounds like you've just volunteered to create ones.

A

Okay, uh what should we do now? Should we I feel like I, have a uh rambled about this for the last 45 minutes.

D

Amazing, it's really interesting and I think what you're, probably um envisioning at the moment and I probably agree- is that it's going to take you a while to do some poking to to make a bit of progress, but it might be something that you're in a position for us to revisit in you know a few days or a week or something when you've made a bit of progress. But again maybe you're you're um you've hit a bit of a wall and you you fancy a bit more pairing, yep.

A

All right, at least uh uh I have a part for like I'll try to like what Jesse mentioned within, and you mentioned like within the wall itself. Can we use the issuing classes to to issue a j oidc token itself, but without going through that entire oauth flow, uh thanks for bearing with my uh incomprehensible rankings.

D

Right so Sharon is, as I say, really interesting, I'm sure we all learned a lot.

C

Yeah and I'll try to I'll take it as my homework to follow up with you on uh sharing the oidc setup. I think the one I set up was a little different, um but uh I think it was for GitHub as an identity provider using oidc and I was very explicit in my instructions to myself on how to decode the JWT. So hopefully I can help you with that.

C

um After this I.

D

Have a a really easy one: actually, that I'd love to I'm hoping someone around the table will actually know the answer without even having to to run any code. Etc.

C

D

um I won't yeah I if I need to in a minute. I will um I have far too much stuff opened.

B

D

Yeah, if, if you don't mind, um I popped the the Mr in here, so if you go to to the overview, you'll have to find the relevant discussion, but there's one um probably one of the unresolved ones. Talking about um this one here, I think it is so we've got a test which is essentially.

D

um Comparing like a web hook payload to the actual data, if that makes sense, and can you see here that it's failing because the customer relations contacts created it is like I, don't know what they are nanoseconds micro seconds different from each other.

D

um So the way that we got around that um I say we as helping someone on my old team. Put this together um initially was to hard code, the date as just a date, without a time that obviously gets rid of the time portion and means that there's no no potential for any kind of discrepancy.

C

um This was the suggestion you applied well.

D

The opposite, so we had it working, um but um the reviewer maintainer weren't overly happy with the hard-coded date. So they said. Why have you got that you shouldn't need that, remove it and weirdly it worked when they ran the test in their environment, but it didn't work in our environment and it didn't work in CI. So we've now come up with another solution. If you want to check the changes tab, um but what I'd love to know is is sort of.

D

If anyone else has experienced this and if the assumption that we came up with so the solution, we came up with.

D

To online 57 now we are doing a contact.reload and that fixed it. So my here's, my completely Bonkers Theory, that's probably yeah wildly inaccurate, but my belief was maybe that the Precision in memory of a timestamp is different to the Precision in the database of a timestamp and therefore, if we don't do the Dot reload we're using the in-memory timestamp um to compare to. If we do the reload we're actually reloading the record, that's been persisted to the database and we've got the the same timestamp, but tell me: has anyone experienced this problem before?

D

Is that the reason is there a better solution? I know there's this like you can freeze time potentially, but.

B

That was going to be my.

D

Comment: do you think that that's a better solution, like I, usually find flakiness intestinal time freezing? It is usually the best way to go. Yeah, okay,.

B

I, am um this definitely Rings a really really distant Bell?

B

For me, this issue that you're talking about and I think I've encountered it years ago and I think what it is is that the if I remember correctly- and it may not be true anymore, but the the time stamp for any of those like created at type Fields- it's stored with the same Precision on the model, but it stood at the Ruby time that the save is made on the model where so it doesn't when when it does the create in the database, it doesn't reload then then respond with whatever the database said for the created time.

B

It's just what was the time at run time in Ruby land when I said save so I think, that's probably what you're seeing there.

D

So almost like you, your model has uh created an updated timestamp, but the database actually ignores what you pass in and just uses whatever the current date.

B

Yeah I think usually that's um default. Now at the leave gotcha.

D

B

So I think I think the model probably doesn't send the time, but it kind of like to Short Circuit having to reload the whole model. Again it just says: oh, you know I'm just going to say that it's now and then, when you reload again, it will be a few milliseconds different, because the database in the.

D

User, so that makes more sense than my theory about the Precision being different, which I couldn't back up um yeah. The only the only thing that would be um interesting to understand here is in this test, and maybe you could actually open up the the the test, the spec itself, that issue Builder um I. Think it's see this one.

D

This one I think is fine, which again we might ask question. Why is that fine? It's the issue Builder spec, that was causing the problem. I'm.

C

Not seeing any other specs in the different now.

D

C

One: okay, oh! This is what we're looking.

D

C

D

So if you so see that there's an expect for labels, so what I'm intrigued by is. Why is that not um throwing the same, assuming that the label has a type that is an updated stamp? Maybe we could check that in the rails, console or or something quickly, just to to have a look but yeah I, I'm I was perplexed. Why we were having that problem with the contact record, but the label record?

D

um No, so I I was also looking at seeing why they're different um and the only thing that we could to identify as being different was that the label is without time zone and the contact is with time zone, um but yeah.

B

C

B

Guess I don't know where.

C

Do we know where this is defined.

D

Yeah I mean it should be an issue mode, but you could just spin up a rouse console and do like issue.lasts.label her captures or something like that and labels.

B

To have created that and updated that yeah in the.

D

B

Oh, that's a good question. Okay.

C

So if my rails console wants to play nicely today, the other thing I was going to say. I just pulled up this issue. I've had this issue Malcolm brought up Ruby versus database time. I've also had an issue where the CI database fails on a test, because it was on a different postgres version and different postgresh versions like have different Precision levels, potentially so.

A

Something like.

C

Ci postgres versus local postgres, which could explain why maybe your machine has a different postgres version from the reviewers machine. That's an idea um this! This recommends not using timed off freeze time freezing but the be like time matcher, which is looking for Less Precision than equality. So that's another idea. The.

D

Challenge with that, you then have to explicitly match on each attribute, which then means you know. It's it's not very dry. You know, if you add a new column to the model, then you've got to update the spec Etc aside.

C

Okay, that makes it okay you're not really you're, just looking for the nested attributes, so.

D

I have an issue chests.

B

D

Have to be dry well yeah. What is more about is wanting to make sure that the test fails if somebody adds add something without having to update the test. If that kind of makes sense, um issue.first I found head labels for me, I, don't know whether you'll be so lucky.

C

D

I did last again oops.

C

C

B

You just grab the last label and get the hook afterwards.

C

Good idea I, so rarely oh, so the first issue has labeled hook adders, but it doesn't have customer relations contacts. Okay, so maybe that's why.

D

um No I, I I think the way specifically of the in our tests we're picking an issue that does have and does have them, but this just confirms that we definitely have a created that and updated uh in our label hook catchers, which adds to my confusion as to. Why would that pass um when we're comparing the label her catchers to the label? So if you you look back in the spec again we're.

C

Doing pretty much exactly.

D

C

Year, no here here.

D

I think where you can only see your.

C

um uh Okay, here we go this one: are we in the right one? You see this? Yes,.

D

So you can see like line 56 and 57 are pretty much identical um and if you look at how we create the labels and how we create the um contacts again we're just using a let it be um yeah. So that's that's what I was really struggling with, you know to say well, what's what's different and the only difference I can find is one stored in the database with time zone ones without so um I. Don't know whether it's.

C

I'm trying to create a contact.

D

Yeah yeah awesome.

C

Okay, oh hold on project.

B

um There might actually be a Precision difference just.

A

B

Between uh what rail sends and what postgres stores I'm just reading the blog post here, okay.

D

B

Says when rails models are saved to the database, any time stamps they have stored using a type in postgres called timestamp without time zone has a micro second resolution. I.E 6 digits after the decimal place.

D

B

uh So when you send a seven digit after the decimal place, timestamp, which I think the is.

D

A

Some platforms.

B

On some platforms, but not all so I think there's a difference on Mac OS and Linux yeah.

D

So we were running on Linux and I. Guess the eyes run on Linux, but almost certainly the the reviewer or maintainer would have probably been on Mac OS. So that's.

B

Interesting so so they've run a test here where they say 70 of the times on Linux had seven digits of precision. So even that wasn't right slightly random.

D

We're losing you a bit now.

C

Oh yeah I, don't know if that was me, I think Malcolm.

B

C

D

Too much training data being transmitted.

B

To the AI, but.

B

C

You're still doing this with their voice.

C

Do one more test see if we can hear you think you're done, let.

B

Me just send that, let me send it.

D

I stuck to a camera, oh.

B

D

That might have improved the the audio without video.

B

I think you'll probably have to come here and rewire my internet to improve it, but.

C

uh This is the issue.

D

Did it say anything on this article but as I'm skimming it that there's a difference when you set it like with time zone without time zone or.

C

Oh because you're wondering why the two attributes are having issues the same way, yeah.

D

B

Long story short.

C

Oh man I'm still losing you. We we get like one interesting.

D

C

It's a bummer I'm.

D

Wondering if the time zone is is takes up part of the Precision. Maybe maybe that's why, if you know I mean so, if you imagine that the field size is the same for with and without time zone, but one has to store the time zone in one dozen then.

B

That might be enough.

C

To push it over.

D

C

Yeah- and you can't see this Fair locally, so you can't look at it and see that.

D

Way, well, it won't for me um it doesn't get pogged, which is is where the the guy in mind our team sort of um been working, I, see, I, see and in CI fails as well, um but the problem is I. Don't know that we would be able to die of that deep under the hood anyway and really really see what was happening. I mean. Maybe you can, but you know, I mean to see what's happening between the model and the database like that. That's active record magic there right so yeah.

C

But you could, you could get the output of the failure or like have them put out what they expect and what they're getting and.

D

Just well, that's that's already on the Mr anyway. So um that's what we were looking at um a minute ago, where we could see that you know. There's a few I, don't know: okay,.

C

um Okay, well, look at that. It sounds like you have volunteered to update our docs on time zones so.

D

I look forward awesome. Let me um just make sure I didn't close the window with that link that Malcolm shed um so um yeah. It's it's really just that that bit about what's the difference between the labels and the contact, um the only way I can think of of being figuring this out, which is a real pain in the backside, but would be to write a local migration to change the time stamp on the contacts from with time zone to without time zone and see.

D

If that then fixes it, um because that that's the only difference I can see between labels and contacts, the dates stored with time zone for one and without the other yeah.

C

I will say that the we should go.

B

C

Favorite way to solve this problem when I was less wise than I am now was just to make every time zone B dot beginning of hour. So it's a nice round time.

D

um Which was essentially our coding dates without time solution, um but sort of. Understandably, it got flagged. My I I I think if we had struggled any further I would have said well, let's, let's not our community contributors problem um like let's get it merged, and then someone can follow up on that, because this is, you know way above even my pay grade that Learners. So but it's interesting awesome, no Wicked! Thanks all for uh indulging me and uh yeah I'll feed back.

C

Awesome. Thank you thanks um both of you for bringing fun issues to discussed. This was great.

B

C

A great day.

B

C